[thelist] robots and the whitehouse.gov site

Brian Cummiskey Brian at hondaswap.com
Mon Feb 2 01:02:14 CST 2004

Stumbling on google for some robot info, I came to the whitehouse.gov 
robot file [1].

They disallow a LOT of directories.  I don't know how large the site 
really is, but it's a ton of 1st level directories.

My question is, why would anyone ever want to disallow their content 
from being indexed?  I can understand the want to disallow 
yourdomain.com/personalstuff or something like that, but why pretty much 
all your content?


[1] - http://www.whitehouse.gov/robots.txt

More information about the thelist mailing list