[thelist] Spider Attack

Tue Mar 15 22:25:29 CST 2005

BMP wrote:

>I need someone to advise me about preventing robot spiders from using excess bandwidth on my website.  The biggest abuser seems to be Mozilla Gecko. Any help would be greatly appreciated.
>
OK, I'll bite.  Mozilla Gecko is not a spider, but currently the most 
popular rival browser on the market.  In some geek circles it has become 
#1.  If you're seeing lots of Gecko in your logs/stats, that is human 
traffic, not spiders.
http://www.mozilla.org/

That said, here's an article that may get you going "How to block 
spambots, ban spybots, and tell unwanted robots to go to hell" by Mark 
Pilgrim.  Some additional links in the comments to the article.
http://url123.com/25633