Brian Cummiskey wrote: > Stumbling on google for some robot info, I came to the whitehouse.gov > robot file [1]. > > They disallow a LOT of directories. I don't know how large the site > really is, but it's a ton of 1st level directories. > > My question is, why would anyone ever want to disallow their content > from being indexed? I can understand the want to disallow > yourdomain.com/personalstuff or something like that, but why pretty > much all your content? > > thoughts? You can read a discussion about it here: http://yro.slashdot.org/article.pl?sid=03/10/27/2052228&mode=thread&tid=103&tid=126&tid=95&tid=99 Edwin Martin -- http://www.bitstorm.org/