[thelist] Spider Attack

BMP microme_2000 at yahoo.com
Tue Mar 15 19:36:16 CST 2005

I need someone to advise me about preventing robot spiders from using excess bandwidth on my website. I know something about HTML, JS but Perl, etc. are not known. I have a robots.txt file, but I am not sure if the spiders searching my site are friendly or not, so I am hesitating about excluding them until I can get more information about them. The biggest abuser seems to be Mozilla Gecko. Any help would be greatly appreciated.

Do you Yahoo!?
 Make Yahoo! your home page   

More information about the thelist mailing list