[thelist] robots and the whitehouse.gov site

[John C Bullas]

At 04:22 03/02/2004, Damien COLA wrote
>Because if the site has no commercial interest in having more traffic,
>they'd want people to find only the start page and look for information
>with the embed navigation.

ALSO

If you have lots of stuff of only "local" or peripheral interest the 
spiders crawling
it use up YOUR bandwith and server processor time unnecessarily

...and all the other reasonss too

NEVER rely on robots.txt to restrict access, see how bany "bots" NEVER read 
robots.txt to
confirm this :(

FB


>-----Original Message-----
>My question is, why would anyone ever want to disallow their content
>from being indexed?  I can understand the want to disallow
>yourdomain.com/personalstuff or something like that, but why pretty much
>
>all your content?
>
>--
>* * Please support the community that supports you.  * *
>http://evolt.org/help_support_evolt/
>
>For unsubscribe and other options, including the Tip Harvester
>and archives of thelist go to: http://lists.evolt.org
>Workers of the Web, evolt !


     *************** John C Bullas **************

    weblog: http://freeroller.net/page/johnbullas/Weblog

**** Eudora Mail 6.0.1.1 / McAfee Virus Scan 4.5.1 ****