[thelist] Identify a Web Crawler's request

Greg Holmes greg.holmes at gmail.com
Tue Jul 6 19:39:20 CDT 2004


Originally, David Travis wrote:
>I am working on a site, which requires IE6. In order to
>prevent users who work with other browsers from accessing
>the site I wrote some kind of filter to check the user
>agent string, and redirect the user to an upgrade-your-browser
>page. This redirection also causes requests from web-crawlers
>(search engines) to be redirected to this page.
>
>The site contains a lot of content, which I want to be
>added to the search engines' indexes.

Then, in response to feedback, he wrote:

>For those who took my question seriously, thank you, you
>helped me a lot!

The fact that your question was not narrowly answered
in the format you asked it does not mean that it was not
taken seriously.

>For all the rest... I am sorry, but no one can ignore the
>fact that IE rules! Over 90% use IE (from ver 4 to 6).

Nobody is ignoring it; they're asking why you need
to screen out other browsers.

>I am aware of all security issues, and performance and
>blah... blah... blah...

Apparently not, if you think those issues are "blah, blah
..."

>but the bottom line is that the user does
>not care about it and USES internet explorer...

Not all users.  And you're right; they don't care what
*your* choice of browser is; they'll just leave if you tell
them to "upgrade" a Volvo to a Gremlin.

>I created a site for internal uses, where I can set "system
>requirements" for browsing it.

Wait a minute; you originally wrote:
>The site contains a lot of content, which I want to be
>added to the search engines' indexes.

Which is it?  Internal uses, or content-rich for search
engines?

>I guess there are better browsers, but for this project I
>simply don't care...

Well, you came to a forum where most people do care.
What did you expect the reaction to be?  But more
importantly, we know that it isn't necessary to restrict
your site building to one browser.  You asked the equivalent
of "how do I do something that almost never makes
sense?"  So forgive us if we have a few questions first ...

I understand that it isn't a very nice feeling, to feel
as if your question is being picked apart.  But you can
now bite your tongue, grit your teeth, and learn from
the experience, or you can lash out.  Up to you.

Greg Holmes


More information about the thelist mailing list