[thelist] spiders spoofing MSIE
Alex Beston
alex at deltatraffic.co.uk
Wed Jun 16 17:03:36 CDT 2004
this might be of help :)
http://www.robotstxt.org/wc/active/html/index.html
Alex
>
>
> I'm working on a project where one sub task is to filter out automated
> agents by their USER_AGENT string. I've done a fair bit of research
> and have come up with a few good lists of bot USER_AGENT strings but I
> want to ensure that the data is as accurate as possible. I
> particularly want to make sure that I'm not missing any agents that
> pretend to be MSIE as these are much harder to pick up using regexp's.
>
> The following is what I've been able to glean from Googling, does
> anyone know of any more?
More information about the thelist
mailing list