[thelist] Spider Attack

Mark Mandel mark.mandel at gmail.com
Tue Mar 15 20:36:06 CST 2005

It could also be a referred spider.

Those are nasty creatures.

If you have Apache, you can user .htaccess to stop them from entering your site.
If not you can just be nasty - like so:

Search spiders are good.

Spiders don't tend to take up too much bandwitdth anyway - all they
grab are html files, which shouldn't be huge in the 1st place.



On Wed, 16 Mar 2005 13:10:14 +1100, Michael Pemberton
<evolt at mpember.net.au> wrote:
> BMP wrote:
> > I need someone to advise me about preventing robot spiders from using excess bandwidth on my website. I know something about HTML, JS but Perl, etc. are not known. I have a robots.txt file, but I am not sure if the spiders searching my site are friendly or not, so I am hesitating about excluding them until I can get more information about them. The biggest abuser seems to be Mozilla Gecko. Any help would be greatly appreciated.
> Mozilla Gecko?
> This is a valid browser.  It is most likely these hits are your visitors
> that are using Mozilla or Firefox.  Do not make any attempt to block
> this user agent.
> --
> Michael Pemberton
> evolt at mpember.net.au
> --
> * * Please support the community that supports you.  * *
> http://evolt.org/help_support_evolt/
> For unsubscribe and other options, including the Tip Harvester
> and archives of thelist go to: http://lists.evolt.org
> Workers of the Web, evolt !

E: mark.mandel at gmail.com
W: www.compoundtheory.com
ICQ: 3094740

More information about the thelist mailing list