[thelist] robots.txt ???
M. Seyon
evoltlist at delime.com
Tue Feb 22 08:27:57 CST 2005
Message from Mike (2/22/2005 08:58 AM)
>At 01:44 22.02.2005 -0800, you wrote:
>>It's definitely worthwhile to create a robots.txt file to specify pages
>>and directories you don't want crawled by the bots. The major search
>>bots (google, msn, etc.) will respect this file and you can save yourself
>>from significant (non-human) server load by using it.
>
>A drawback is that someone can get a list of all the folders you
>don't want to be seen, simply by pointing their browser to the
>robots.txt file :P
Unless you have directory browsing enabled this shouldn't be a problem with
casual surfers though.
What I question is that as far as I know robots aren't obliged to obey the
instructions in robots.txt files. "Well intentioned" ones like Google's or
other search engine indexers probably will but malicious ones - email
harvesters, etc probably won't.
regards.
-marc
--
Trinidad Carnival in all its photographic glory. Playyuhself.com
http://www.playyuhself.com/
More information about the thelist
mailing list