[thelist] robots.txt ???

[M. Seyon]

Message from Mike (2/22/2005 08:58 AM)
>At 01:44 22.02.2005 -0800, you wrote:
>>It's definitely worthwhile to create a robots.txt file to specify pages 
>>and directories you don't want crawled by the bots.  The major search 
>>bots (google, msn, etc.) will respect this file and you can save yourself 
>>from significant (non-human) server load by using it.
>
>A drawback is that someone can get a list of all the folders you
>don't want to be seen, simply by pointing their browser to the
>robots.txt file   :P

Unless you have directory browsing enabled this shouldn't be a problem with 
casual surfers though.

What I question is that as far as I know robots aren't obliged to obey the 
instructions in robots.txt files. "Well intentioned" ones like Google's or 
other search engine indexers probably will but malicious ones - email 
harvesters, etc probably won't.

regards.
-marc

--
Trinidad Carnival in all its photographic glory. Playyuhself.com
http://www.playyuhself.com/