[thelist] Re: Identify a Web Crawler's request

[Michael Harrington]

David,

I was thinking of preventing the Web crawlers from crawling your "upgrade-your-browser page" and stopping there. I had the same problem until I put
the <meta NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> on my "upgrade-your-browser page". This page would get indexed and the crawlers would stop at it
but not now. You do not need to identify every Web crawler out there with this tag.

If I understood your question correctly, your problem is with the Web crawlers and your "upgrade-your-browser page" stopping the crawling the rest of
your site. Is this why you asked "How do I identify a request from a web-crawler?"

(David wrote: "This redirection also causes requests from web-crawlers (search engines) to be redirected to this page.")

Michael

thelist-request at lists.evolt.org wrote:

>
>
> Date: Wed, 7 Jul 2004 09:53:35 +0200
> From: "David Travis" <dwork at macam.ac.il>
> To: <thelist at lists.evolt.org>
> Subject: RE: [thelist] Re: Identify a Web Crawler's request
> Message: 26
>
> Hi Michael,
>
> Thanks for your reply. In my case it is simply useless to browse the site
> unless you meet the minimum requirements, since it relies heavily on the
> client's browser.
>
> Thanks,
> David.
>