[thelist] robots and the whitehouse.gov site

Frank lists at frankmarion.com
Tue Feb 3 01:50:14 CST 2004

At 02:02 AM 2004-02-02, you wrote:
>My question is, why would anyone ever want to disallow their content from 
>being indexed?  I can understand the want to disallow 
>yourdomain.com/personalstuff or something like that, but why pretty much 
>all your content?

To quote any white house official: "911".

Actually, it may be security. Some might be image folders, some might be 
folders with executables, or what-not necessary to keep the site up and 
running. Some might be hacker traps. There are a lot of reasons.

I personally have a tendency to place an index document with a re-direct to 
the main page in all my folders but the top one. You can access any of the 
URLs if you know them, or follow the link, but you can't just start 
snooping though the site's structure.

Also, if it's a dynamic app, there may be documents that require that you 
set up some conditions before being able to access it, such as cookies, or 
session variables, by making sure the user flows though the documents that 
set them up, the reduce the possibilities of errors occurring on the site.

My 2 cents.

Frank Marion     lists at frankmarion.com      Keep the signal high.  

More information about the thelist mailing list