[thelist] robots.txt ???

David Siedband technique at oceanicsky.com
Tue Feb 22 03:44:01 CST 2005


It's definitely worthwhile to create a robots.txt file to specify pages 
and directories you don't want crawled by the bots.  The major search 
bots (google, msn, etc.) will respect this file and you can save 
yourself from significant (non-human) server load by using it.

--
David Siedband
generation-xml.com






On Feb 22, 2005, at 12:45 AM, D Ross wrote:

> I've just recently learned about putting a robots.txt file in the 
> directory/root. I'm not real familiar with it and what to do or if 
> it's actually a good thing or a waste of time. What do you folks 
> think?



More information about the thelist mailing list