[thelist] Identify a Web Crawler's request

Daniel Champion dchampion at clacks.gov.uk
Tue Jul 6 08:14:31 CDT 2004

> How do I identify a request from a web-crawler?

They will identify themselves with a user-agent string like any browser. 
For a list see:


Daniel Champion - Web Dev Mgr - Clackmannanshire Council
e: dchampion at clacks.gov.uk    t: 01259 452258
f: 01259 452265    w: http://www.clacksweb.org.uk

This email and any attachments have been scanned for viruses prior to leaving Clackmannanshire Council.

Clackmannanshire Council will not be liable for any losses as a result of viruses being passed on.


More information about the thelist mailing list