[thelist] robots and the whitehouse.gov site
John C Bullas
jcbullas at nildram.co.uk
Tue Feb 3 01:12:53 CST 2004
At 04:22 03/02/2004, Damien COLA wrote
>Because if the site has no commercial interest in having more traffic,
>they'd want people to find only the start page and look for information
>with the embed navigation.
ALSO
If you have lots of stuff of only "local" or peripheral interest the
spiders crawling
it use up YOUR bandwith and server processor time unnecessarily
...and all the other reasonss too
NEVER rely on robots.txt to restrict access, see how bany "bots" NEVER read
robots.txt to
confirm this :(
FB
>-----Original Message-----
>My question is, why would anyone ever want to disallow their content
>from being indexed? I can understand the want to disallow
>yourdomain.com/personalstuff or something like that, but why pretty much
>
>all your content?
>
>--
>* * Please support the community that supports you. * *
>http://evolt.org/help_support_evolt/
>
>For unsubscribe and other options, including the Tip Harvester
>and archives of thelist go to: http://lists.evolt.org
>Workers of the Web, evolt !
*************** John C Bullas **************
weblog: http://freeroller.net/page/johnbullas/Weblog
**** Eudora Mail 6.0.1.1 / McAfee Virus Scan 4.5.1 ****
More information about the thelist
mailing list