[thelist] Spider Attack

BMP microme_2000 at yahoo.com
Tue Mar 15 19:36:16 CST 2005


Hi
 
I need someone to advise me about preventing robot spiders from using excess bandwidth on my website. I know something about HTML, JS but Perl, etc. are not known. I have a robots.txt file, but I am not sure if the spiders searching my site are friendly or not, so I am hesitating about excluding them until I can get more information about them. The biggest abuser seems to be Mozilla Gecko. Any help would be greatly appreciated.
 
Thanks,
Mike

		
---------------------------------
Do you Yahoo!?
 Make Yahoo! your home page   


More information about the thelist mailing list