[thelist] Identify a Web Crawler's request

Daniel Champion dchampion at clacks.gov.uk
Tue Jul 6 08:14:31 CDT 2004


> How do I identify a request from a web-crawler?

They will identify themselves with a user-agent string like any browser. 
For a list see:

http://www.psychedelix.com/agents.html

HTH
-- 
Daniel Champion - Web Dev Mgr - Clackmannanshire Council
e: dchampion at clacks.gov.uk    t: 01259 452258
f: 01259 452265    w: http://www.clacksweb.org.uk



**********************************************************************
This email and any attachments have been scanned for viruses prior to leaving Clackmannanshire Council.

Clackmannanshire Council will not be liable for any losses as a result of viruses being passed on.

www.clacksweb.org.uk
**********************************************************************



More information about the thelist mailing list