[thelist] robots and the whitehouse.gov site

Wil Cone wil at inaridesign.com
Tue Feb 3 01:24:39 CST 2004


According to some, it's the WH's attempt to block Google caching, so
text changes aren't easy to discover...

http://shock-awe.info/archive/000965.php

- Wil

> -----Original Message-----
> From: thelist-bounces at lists.evolt.org 
> [mailto:thelist-bounces at lists.evolt.org] On Behalf Of Brian Cummiskey
> Sent: Monday, February 02, 2004 1:02 AM
> To: thelist
> Subject: [thelist] robots and the whitehouse.gov site
> 
> 
> Stumbling on google for some robot info, I came to the whitehouse.gov 
> robot file [1].
> 
> They disallow a LOT of directories.  I don't know how large the site 
> really is, but it's a ton of 1st level directories.
> 
> My question is, why would anyone ever want to disallow their content 
> from being indexed?  I can understand the want to disallow 
> yourdomain.com/personalstuff or something like that, but why 
> pretty much 
> all your content?
> 
> thoughts?
> 
> [1] - http://www.whitehouse.gov/robots.txt



More information about the thelist mailing list