[thelist] Crawler Index

Tony Grimes info at scribenewmedia.com
Wed Jul 9 01:40:47 CDT 2003


Hey all,

We're trying to weed out all of the bot and crawler activity from our log
files. Are there any up-to-date, downloadable indexes of known bots/crawlers
with User_Agent strings?

I've found a couple on Google, but they're either out of date, incomplete or
aren't in an easily downloadable format. I also found this one in thelist
archives:

http://www.spiderkb.com/spider_id.cfm

Does anyone know if this list is still being maintained (I think Jeff posted
it originally)?

TIA,

Tony Grimes
Senior Project Manager, Scribe NewMedia
Email: info at scribenewmedia.com
http://www.scribenewmedia.com/



More information about the thelist mailing list