[thelist] Common spider/bot USER_AGENT strings?

[George Dillon]

I'm currently using XSSI to prevent hits from spiders showing in my
CGI-generated logs something like this:

<!--#if expr="($HTTP_USER_AGENT =/Atomz/)
 || ($HTTP_USER_AGENT =/Mercator/)
 || ($HTTP_USER_AGENT =/bumblebee/)
 || ($HTTP_USER_AGENT =/pider/)
 || ($HTTP_USER_AGENT =/rawler/)"
 --><!--#else

 --><!--#exec cgi="/cgi-bin/logger.cgi"

 --><!--#endif -->

Is there a list anywhere of common spider/bot USER_AGENT strings I could add
to this?

And/or is there a better way of doing what I'm doing using XSSI i.e. neater
syntax or more reliable method?

TIA

George Dillon