[thelist] robots and the whitehouse.gov site

Damien COLA damiencola at wanadoo.fr
Mon Feb 2 22:22:37 CST 2004

Because if the site has no commercial interest in having more traffic,
they'd want people to find only the start page and look for information
with the embed navigation.

-----Original Message-----
My question is, why would anyone ever want to disallow their content 
from being indexed?  I can understand the want to disallow 
yourdomain.com/personalstuff or something like that, but why pretty much

all your content?

More information about the thelist mailing list