[thelist] Spider Attack

Michael Pemberton evolt at mpember.net.au
Tue Mar 15 20:10:14 CST 2005

BMP wrote:
> I need someone to advise me about preventing robot spiders from using excess bandwidth on my website. I know something about HTML, JS but Perl, etc. are not known. I have a robots.txt file, but I am not sure if the spiders searching my site are friendly or not, so I am hesitating about excluding them until I can get more information about them. The biggest abuser seems to be Mozilla Gecko. Any help would be greatly appreciated.

Mozilla Gecko?

This is a valid browser.  It is most likely these hits are your visitors 
that are using Mozilla or Firefox.  Do not make any attempt to block 
this user agent.

Michael Pemberton
evolt at mpember.net.au

More information about the thelist mailing list