[thelist] robots and the whitehouse.gov site

Stuart Young syoung at unitec.ac.nz
Tue Feb 3 15:25:01 CST 2004


> Brian at hondaswap.com 02/02/2004 20:02:14 >
Stumbling on google for some robot info, I came to the whitehouse.gov 
robot file [1].
They disallow a LOT of directories.  
[1] - http://www.whitehouse.gov/robots.txt 

I haven't checked them all, but all of the URLs that I did check were 404s now, and also half of them are about Iraq! Which tends to add weight to the cover-up conspiracy theories mentioned here.

cheers

Dr Stuart Young,       +64 (0)9-815 4321 Ex 8656
<syoung at unitec.ac.nz>     
Lecturer, School of Computing and Information Technology,
UNITEC Institute of Technology, Auckland, New Zealand
http://hyperdisc.unitec.ac.nz/staff/syoung.htm
I would provide a URL for my official staffpage, but its too long and complex



More information about the thelist mailing list