Distributed bandwidth (was RE: [Theforum] Re: [Content] Mail Alert! - Sat Jun 22 31226)

[.jeff]

beau,

let me preface this reply by saying that william (neuro) and i have talked
at length about this in irc today.  i've been listening to all the
conversations about this issue and have most of a data model together that
will address most everything.

what i'm proposing is a long-term solution for beo.  william has already
come up with something that can be used as a short term solution but will
require mirrors to have a complete copy of the archive.  aside from those
mirrors getting their complete copies in place, this solution can be
implemented immediately to take off some of the weight in the short-term.

[more inline]

><><><><><><><><><><><><><><><><><><><><><><><><><><><><><
> From: Beau Hartshorne
>
> OK, so what about two or three whole-archive mirrors,
> then a bunch of others that host one or several
> individual browsers?
><><><><><><><><><><><><><><><><><><><><><><><><><><><><><

a mirror can host one or more browsers.  this allows us to setup a source
like mozilla.org as a mirror for each of the various versions of mozilla.
if a mirror wants to handle all browsers in the archive, they'll first need
to get a copy of all the browsers and then get setup as a mirror for all
browsers.  i think this is the most reasonable approach since most mirrors
will only mirror a few browsers and very few will be entire archive mirrors.

><><><><><><><><><><><><><><><><><><><><><><><><><><><><><
> How many actual files need to be mirrored?
><><><><><><><><><><><><><><><><><><><><><><><><><><><><><

good question.

[digs through theforum archives]

i found kb listings for the top-level folders, but no information on the
total number of files.  dean, can you chime in with some figures for us?

><><><><><><><><><><><><><><><><><><><><><><><><><><><><><
> > Suggestion also for if we have multiple options for
> > download: rotate the order of the links since people
> > tend to pick options towards the top of a list (yes,
> > I know, sensible people will pick the nearest mirror,
> > but the two factors operate separately)
>
> Should we try to regulate how many hits go to each
> mirror, and remove the mirror from the list once a
> certain number of hits is reached (to keep someone
> from inadvertently "donating" too much bandwidth)?
><><><><><><><><><><><><><><><><><><><><><><><><><><><><><

actually, i can go one step further.  when a mirror is setup you can
optionally give it a max amount of transfer in a given time period (probably
a month).  each time a browser is downloaded the download will be recorded
and the weight of the file will be deducted from the mirrors available
transfer for that month.  if a mirror has its bandwidth used up in the month
it will disappear from the list of available mirrors.  once that month is
over it will once again reappear in the list of mirrors.

in cases where a browser has multiple mirrors assigned, the mirrors will be
sorted from the mirror with the most available on top to the mirror with the
least still available on the bottom.

mirrors that don't have an optional data transfer limit defined will always
be at the top of the list.  these will almost always be links to the
original site where the browser was released/available.  i would hope that
we could define an unlimited data transfer mirror for each of the browsers
in the archive so we can be sure that any browser can be downloaded at any
time.

one quick note if it hasn't been made clear yet.  i don't want *any* of the
downloads for browsers coming off of evolt.org servers unless we're in a
financial position to support it.  even then, i'd prefer to keep it as a
source available only for those interested in offering mirror services to
get the browser or browsers they wanted to mirror for the archive that
weren't currently mirrored anywhere else.  our copy of the archive would sit
behind some form of login or accessible via http or anonymous ftp at all.
our copy would act as the master copy.  if a mirror finds and makes
available a browser we don't have in our master copy, someone (hopefully the
mirror) would let us know and we'd grab a copy to put in our private
collection just in case the mirror should ever go offline and not come back.
this could be facilitated by the new beo system not allowing mirrors to
create new browser entries in our database directly or if directly not
without approval from one of the subgroups (content?).  one of the criteria
for approval would be releasing a copy of the newly submitted browser for
our master archive.

i'm going to expand on this whole mirroring idea alittle more so this email
can be used as a rough-spec of sorts when it comes time to develop this
solution.

currently the browser archive is only browsable by finding the manufacturer
and then drilling down through os options and then version options to the
browser you want.  since all (within reason) the information about a browser
will be stored in a database we will be able to offer users other ways of
browsing the archive.  we could let them choose os, then manufacturer, then
version.  we could let them choose to view the top downloads starting with
the one with the most to date.  we could let them drill down as they do
currently.  if the data is available, we could let them browse the list of
browsers by release date -- newest to oldest and vice versa.  how about
viewing the browsers from largest to smallest file size or vice versa?

we could also very easily offer a search to find a browser by a particular
name.

once the user finds the browser they want i imagine the following steps
necessary to download it.

 - click a link to download the browser
   which takes them to a list of mirrors
 - the user selects the mirror they want
   to download it from which takes them
   to a page giving information about
   the mirror (which could be controlled
   by the mirror, much like a user profile)
 - this mirror information page would then
   meta refresh to a simple server-side
   download counting page which would also
   do a location change to the actual file
   the user wanted to download

before anybody comments on too many clicks, let me say that the user only
has to click twice, once to get to the list of mirrors, a second time to
download from a mirror.  the other two location changes would happen without
the user even noticing they took place.  they'd simply land on the mirror
profile page and the download "save file" prompt would pop up.

there are other possible feature ideas as well.

 - we could implement comments and ratings for browsers.
 - we could implement comments and ratings for mirrors.
 - we could have a scheduled task that would run nightly
   and check each of the mirrors for availability.  if a
   mirror isn't available when the scheduled task runs it
   would be temporarily deactivated.  if it's a member's
   mirror an email could be sent to the email address on
   record for that mirror alerting the member.  in all
   cases an email would go to one of the subgroups (desdev?
   content?  sysadmin?) alerting them to the situation with
   that mirror.  this is especially important where we only
   have a manufacturer's link to download a particular
   browser and the manufacturer's site goes offline.  we'd
   then have to make a decision about where to put the our
   archived copy of that browser so the link to download it
   could be restored.

these are just some ideas.  i don't mean for them to get in the way of
development of the core pieces noted earlier in this email, but i thought
it'd be fun to explore some of the fun things we could do with this down the
road.

anybody have any input they'd like to put in to this?

thanks,

.jeff

http://evolt.org/
jeff at members.evolt.org
http://members.evolt.org/jeff/