[thelist] robots.txt

Tony Crockford tonyc at boldfish.co.uk
Sun Nov 16 18:27:11 CST 2003


On Sun, 16 Nov 2003 15:44:57 -0800, Diane Soini <dianesoini at earthlink.net> 
wrote:

> How well is the robots.txt file obeyed?

http://diveintomark.org/archives/2003/02/26/how_to_block_spambots_ban_spybots_and_tell_unwanted_robots_to_go_to_hell

And this might help too, sorry - it was the next link unfollowed before I 
hit send....

hth

Tony


More information about the thelist mailing list