[thelist] Identify a Web Crawler's request

David Travis dwork at macam.ac.il
Tue Jul 6 05:56:00 CDT 2004


Hi All,

Interesting question.

I am working on a site, which requires IE6. In order to prevent users who
work with other browsers from accessing the site I wrote some kind of filter
to check the user agent string, and redirect the user to an
upgrade-your-browser page. This redirection also causes requests from
web-crawlers (search engines) to be redirected to this page.

The site contains a lot of content, which I want to be added to the search
engines' indexes.

Now to the question: How do I identify a request from a web-crawler? Is
there a standard header in the HTTP Request to check? I am particularly
interested in Google's headers since it is most popular.

Thanks in advance,
David.




More information about the thelist mailing list