[thelist] Avoiding logging of bots

[Max Schwanekamp]

> From: M. Seyon 
> >robots in their multitudes
> Can't you just use a robots.txt exclusion file to block 
> access to that 
> particular file?
> http://www.robotstxt.org/wc/exclusion.html
> 
> I don't know how many robots obey it, but most of the 
> well-behaved ones should.

Yes, we do have a robots.txt file in place, but there are lots and lots of
bots that ignore it, some of which do things like spoof their UA string to
look like the Googlebot or Slurp, for example.