[thelist] robots and the whitehouse.gov site

Edwin Martin edwin at bitstorm.org
Tue Feb 3 12:53:18 CST 2004


Brian Cummiskey wrote:

> Stumbling on google for some robot info, I came to the whitehouse.gov 
> robot file [1].
>
> They disallow a LOT of directories.  I don't know how large the site 
> really is, but it's a ton of 1st level directories.
>
> My question is, why would anyone ever want to disallow their content 
> from being indexed?  I can understand the want to disallow 
> yourdomain.com/personalstuff or something like that, but why pretty 
> much all your content?
>
> thoughts?

You can read a discussion about it here:

http://yro.slashdot.org/article.pl?sid=03/10/27/2052228&mode=thread&tid=103&tid=126&tid=95&tid=99

Edwin Martin

-- 
http://www.bitstorm.org/



More information about the thelist mailing list