[thelist] robots.txt ???
Mike
evolt at muinar.com
Tue Feb 22 06:58:31 CST 2005
At 01:44 22.02.2005 -0800, you wrote:
>It's definitely worthwhile to create a robots.txt file to specify pages
>and directories you don't want crawled by the bots. The major search bots
>(google, msn, etc.) will respect this file and you can save yourself from
>significant (non-human) server load by using it.
A drawback is that someone can get a list of all the folders you
don't want to be seen, simply by pointing their browser to the
robots.txt file :P
_____ mike s. krischker http://webdesign-schweiz.ch/
webpro mailing list http://webdesign-list.com/
flashpro mailing list http://flash-list.com/
More information about the thelist
mailing list