Distributed bandwidth (was RE: [Theforum] Re: [Content] Mail Alert! - Sat Jun 22 31226)

William Anderson neuro at well.com
Mon Jun 24 10:47:34 CDT 2002

---- Original Message ----
From: "Beau Hartshorne" <beau at members.evolt.org>
To: <theforum at lists.evolt.org>
Sent: Monday, June 24, 2002 4:22 PM
Subject: RE: Distributed bandwidth (was RE: [Theforum] Re: [Content]
Mail Alert! - Sat Jun 22 31226)

> > imho, rsync is the first and best service to use for mirrorers to
> > grab and update their parts, or all, of the archive.  http and ftp
> > should be 'last resorts'.
> I wouldn't expect someone mirroring only a few files to use rsync. My
> interpretation of the spec is that almost anyone can volunteer. They
> fill out a form (where they include how much bandwidth they can
> donate), download a browser from beo, upload the browser to their
> server, they go back to the beo app to confirm the upload, the beo
> app checks to see if the link is good, then makes the link available.

I would expect as many people as possible to use the most efficient
mirroring tool available - that is rsync.  Whether you are mirroring 2
megabytes, or 2 terabytes, rsync just rocks.  It can be secured over ssh
too.  I would much rather see people using rsync if they can, as opposed to
constantly wget'ing stuff, or - god forbid - manually downloading files.

Automating the mirroring process also helps us on the backend - we only need
to add a file once to beo, and the curator(s) will automatically get that
file they next time they connect.  If there are no changes, they don't fetch
anything.  They don't even do anything like HEAD checks to see if the data
has changed - rsync tells them so.

I know i'm coming off as a bit arrogant sounding saying that, but rsync
really really is the way to go imho.  I maintain a few mirrors here at home,
and one of them is an FTP mirror, maintained over FTP.  To fetch the changes
on a 2.2gig dataset takes a minute or so to download.  When I rsync the same
dataset (albeit locally), I can see the amount of change data changing hands
is minimal compared to that transferred during the FTP session.  (did that
make sense? :)

> [possible bandwidth misuse of mirrors by bad people snippage]
> The only solution I can think of is this: get volunteers to check
> referrers, and only send files to anyone referred by evolt.org. Is
> something like that possible?

Yup, but it really does depend on the config of the curators kit - once we
start getting loads of people mirroring bits or all of beo, we're moving
away from the controlled environment that beo represents, and letting loose
a large amount of heterogenous services.  Some may be on Linux, some NT,
win2K, Tru64, Solaris, etc etc, some may be Apache, others IIS, Roxen, etc


> Does anyone else think this is necessary?

possibly ...

_ __/|   ___  ___ __ _________  "Hell hath no fury like a woman scorned
\`O_o'  / _ \/ -_) // / __/ _ \  for Sega." -- Brodie, 'Mallrats'
=(_ _)=/_//_/\__/\_,_/_/  \___/ @ well.com :: William Anderson
   U - Ack! Phttpt! Thhbbt!     http://neuro.me.uk/

More information about the theforum mailing list