[thelist] robots and the whitehouse.gov site
Brian Cummiskey
Brian at hondaswap.com
Mon Feb 2 01:02:14 CST 2004
Stumbling on google for some robot info, I came to the whitehouse.gov
robot file [1].
They disallow a LOT of directories. I don't know how large the site
really is, but it's a ton of 1st level directories.
My question is, why would anyone ever want to disallow their content
from being indexed? I can understand the want to disallow
yourdomain.com/personalstuff or something like that, but why pretty much
all your content?
thoughts?
[1] - http://www.whitehouse.gov/robots.txt
More information about the thelist
mailing list