[thelist] robots.txt ???

[Mike]

At 01:44 22.02.2005 -0800, you wrote:
>It's definitely worthwhile to create a robots.txt file to specify pages 
>and directories you don't want crawled by the bots.  The major search bots 
>(google, msn, etc.) will respect this file and you can save yourself from 
>significant (non-human) server load by using it.

A drawback is that someone can get a list of all the folders you
don't want to be seen, simply by pointing their browser to the
robots.txt file   :P

_____  mike s. krischker  http://webdesign-schweiz.ch/
        webpro mailing list  http://webdesign-list.com/
        flashpro mailing list  http://flash-list.com/