[thelist] robots.txt ???

[dave [ evolt ]]

Mike said the following on 02.22.2005 7:58 AM:

>> It's definitely worthwhile to create a robots.txt file to specify 
>> pages and directories you don't want crawled by the bots.  The major 
>> search bots (google, msn, etc.) will respect this file and you can 
>> save yourself from significant (non-human) server load by using it.
>
> A drawback is that someone can get a list of all the folders you
> don't want to be seen, simply by pointing their browser to the
> robots.txt file   :P
>
a great example of that is at http://www.whitehouse.gov/robots.txt.

i also noticed that a few sites i set up the robots.txt file for popped 
up in search engines quicker than those without.