Mike said the following on 02.22.2005 7:58 AM: >> It's definitely worthwhile to create a robots.txt file to specify >> pages and directories you don't want crawled by the bots. The major >> search bots (google, msn, etc.) will respect this file and you can >> save yourself from significant (non-human) server load by using it. > > A drawback is that someone can get a list of all the folders you > don't want to be seen, simply by pointing their browser to the > robots.txt file :P > a great example of that is at http://www.whitehouse.gov/robots.txt. i also noticed that a few sites i set up the robots.txt file for popped up in search engines quicker than those without.