[thelist] spiders spoofing MSIE

Alex Beston alex at deltatraffic.co.uk
Wed Jun 16 17:03:36 CDT 2004


this might be of help :)

http://www.robotstxt.org/wc/active/html/index.html

Alex

>
>
> I'm working on a project where one sub task is to filter out automated 
> agents by their USER_AGENT string. I've done a fair bit of research 
> and have come up with a few good lists of bot USER_AGENT strings but I 
> want to ensure that the data is as accurate as possible. I 
> particularly want to make sure that I'm not missing any agents that 
> pretend to be MSIE as these are much harder to pick up using regexp's.
>
> The following is what I've been able to glean from Googling, does 
> anyone know of any more? 

 


More information about the thelist mailing list