[thelist] Identify a Web Crawler's request
Howard Scott
howard at image100.com
Tue Jul 6 08:39:00 CDT 2004
I have to ask...
Why would you want to create a site in this day and age that only worked in
IE6?
Unless of course it's some kind of ASP solution or an intranet type thing?
Howard
--
On 7/6/04 11:56 AM, "David Travis" <dwork at macam.ac.il> wrote:
> Hi All,
>
> Interesting question.
>
> I am working on a site, which requires IE6. In order to prevent users who
> work with other browsers from accessing the site I wrote some kind of filter
> to check the user agent string, and redirect the user to an
> upgrade-your-browser page. This redirection also causes requests from
> web-crawlers (search engines) to be redirected to this page.
>
> The site contains a lot of content, which I want to be added to the search
> engines' indexes.
>
> Now to the question: How do I identify a request from a web-crawler? Is
> there a standard header in the HTTP Request to check? I am particularly
> interested in Google's headers since it is most popular.
>
> Thanks in advance,
> David.
>
More information about the thelist
mailing list