[thelist] robots and the whitehouse.gov site

David Bindel dbindel at austin.rr.com
Mon Feb 2 22:14:01 CST 2004

On Mon, 2004-02-02 at 01:02, Brian Cummiskey wrote:
> My question is, why would anyone ever want to disallow their content 
> from being indexed?  I can understand the want to disallow 
> yourdomain.com/personalstuff or something like that, but why pretty much 
> all your content?

My guess is that they want to severely limit the number of entry points
to their website so that most visitors will enter at the homepage.

I can't really think of a practical reason that would impel me to do
that to one of my own sites, but then again, I'm not a big, famous
entity like the White House.

David Bindel <dbindel at austin.rr.com>

Website Developer

More information about the thelist mailing list