[thelist] robots.txt ???
David Siedband
technique at oceanicsky.com
Tue Feb 22 03:44:01 CST 2005
It's definitely worthwhile to create a robots.txt file to specify pages
and directories you don't want crawled by the bots. The major search
bots (google, msn, etc.) will respect this file and you can save
yourself from significant (non-human) server load by using it.
--
David Siedband
generation-xml.com
On Feb 22, 2005, at 12:45 AM, D Ross wrote:
> I've just recently learned about putting a robots.txt file in the
> directory/root. I'm not real familiar with it and what to do or if
> it's actually a good thing or a waste of time. What do you folks
> think?
More information about the thelist
mailing list