robots.txt (was Re: [theforum] Vesana network)

[Dean Mah]

On Fri, May 21, 2004 at 02:54:14AM -0400, David Kaufman wrote:
> by the way:
> 
> David Kaufman wrote:
> >
> > Elfur Logadóttir wrote:
> >> Does anyone know a thing or two about Vesana Network?
> >
> > ...according to Marlene back in december,
> >
> http://lists.evolt.org/theforumarchive/Week-of-Mon-20031215/000510.html
> 
> 
> which we'd all know what was in our list archive, except that:
> 
> WHY ON EARTH do we *intentionally*, explicitly defeat googling our own
> list archives with this piece of brilliance:
>
> http://lists.evolt.org/robots.txt
> User-agent: *
> Disallow: /thechatarchive/
> Disallow: /adminarchive/
> Disallow: /theforumarchive/
> Disallow: /email-addresses/
> Disallow: /harvest/
> 
> when we don't even have a search engine of our own, with which to search
> our own archives?

I'm guessing because theforum, admin, and thechat were deemed to be
sensitive.  theforum because it dealt with evolt.org's future.
thechat because people say things more freely in thechat that they
consider off the record.  admin because it used to be more like
content and I used to index it using htdig.  email-addresses is
pointless now, I don't think that it exists.  Finally, the harvest, I
just put in because about a week ago, I added the search capability
back to the interface.  Also, since the harvest is a set of Perl
scripts (not mod_perl), spiders were driving the load average up on
the raq to >20 at times.


> is it that we're insane?  or the tips that we harvest some trade secret
> we're planning to patent some day?

Hopefully that has been adequately addressed.


> everyone who's ever been frustrated by looking for, but not finding
> old forum posts using any method evolt doesnt have like a search
> engine, such as google, please reply to this message with a
> gratuitious, "WTF?!?" and then someone please delete this damn
> robots.txt file.

Also, I humbly apologize for harping on the lists hosting post.
Apparently it was not sent to theforum.

Dean