[thelist] Spider Attack
Michael Pemberton
evolt at mpember.net.au
Tue Mar 15 20:10:14 CST 2005
BMP wrote:
> I need someone to advise me about preventing robot spiders from using excess bandwidth on my website. I know something about HTML, JS but Perl, etc. are not known. I have a robots.txt file, but I am not sure if the spiders searching my site are friendly or not, so I am hesitating about excluding them until I can get more information about them. The biggest abuser seems to be Mozilla Gecko. Any help would be greatly appreciated.
Mozilla Gecko?
This is a valid browser. It is most likely these hits are your visitors
that are using Mozilla or Firefox. Do not make any attempt to block
this user agent.
--
Michael Pemberton
evolt at mpember.net.au
More information about the thelist
mailing list