[thelist] User Agent question

[Ken Schaefer]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
From: "Shawn K. Quinn" <skquinn at xevious.kicks-ass.net>
Subject: Re: [thelist] User Agent question


: On Friday 2004 February 27 18:50, Maximillian Schwanekamp wrote:
: > Got a user agent question - I'm setting up a 404 script for my client
: > to redirect traffic according to the url they entered.  For
: > optimization's sake, I would like to skip all processing if the user
: > agent is a spider.
: > ...
: > I am not looking for perfect treatment - if a UA string is spoofed,
: > fine. But when a spider hits the site 50-100 times, there is no reason
: > for the script to be doing a database lookup every time.
:
: Maybe it would be best to not have so many 404 errors to begin with.
: Remember, "URIs don't change; people change them." (from the W3C Style
: Guide)
:
: -- 
: Shawn K. Quinn
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In addition to Shawn's comments, I would add that an appropriate server-side
caching mechanism would probably alleviate the load on your DB server.

Many server-side programming environments would have something you could use
to cache the results of database lookups.

Cheers
Ken