[thelist] Spider Attack
Mark Mandel
mark.mandel at gmail.com
Tue Mar 15 20:36:06 CST 2005
It could also be a referred spider.
Those are nasty creatures.
If you have Apache, you can user .htaccess to stop them from entering your site.
If not you can just be nasty - like so:
http://www.compoundtheory.com/?action=displayPost&ID=48
Search spiders are good.
Spiders don't tend to take up too much bandwitdth anyway - all they
grab are html files, which shouldn't be huge in the 1st place.
HTH
Mark
On Wed, 16 Mar 2005 13:10:14 +1100, Michael Pemberton
<evolt at mpember.net.au> wrote:
> BMP wrote:
> > I need someone to advise me about preventing robot spiders from using excess bandwidth on my website. I know something about HTML, JS but Perl, etc. are not known. I have a robots.txt file, but I am not sure if the spiders searching my site are friendly or not, so I am hesitating about excluding them until I can get more information about them. The biggest abuser seems to be Mozilla Gecko. Any help would be greatly appreciated.
>
> Mozilla Gecko?
>
> This is a valid browser. It is most likely these hits are your visitors
> that are using Mozilla or Firefox. Do not make any attempt to block
> this user agent.
>
> --
> Michael Pemberton
> evolt at mpember.net.au
>
>
> --
>
> * * Please support the community that supports you. * *
> http://evolt.org/help_support_evolt/
>
> For unsubscribe and other options, including the Tip Harvester
> and archives of thelist go to: http://lists.evolt.org
> Workers of the Web, evolt !
>
--
E: mark.mandel at gmail.com
W: www.compoundtheory.com
ICQ: 3094740
More information about the thelist
mailing list