[thelist] robots and the whitehouse.gov site
Frank
lists at frankmarion.com
Tue Feb 3 01:50:14 CST 2004
At 02:02 AM 2004-02-02, you wrote:
>My question is, why would anyone ever want to disallow their content from
>being indexed? I can understand the want to disallow
>yourdomain.com/personalstuff or something like that, but why pretty much
>all your content?
To quote any white house official: "911".
Actually, it may be security. Some might be image folders, some might be
folders with executables, or what-not necessary to keep the site up and
running. Some might be hacker traps. There are a lot of reasons.
I personally have a tendency to place an index document with a re-direct to
the main page in all my folders but the top one. You can access any of the
URLs if you know them, or follow the link, but you can't just start
snooping though the site's structure.
Also, if it's a dynamic app, there may be documents that require that you
set up some conditions before being able to access it, such as cookies, or
session variables, by making sure the user flows though the documents that
set them up, the reduce the possibilities of errors occurring on the site.
My 2 cents.
--
Frank Marion lists at frankmarion.com Keep the signal high.
More information about the thelist
mailing list