[thelist] robots.txt ???

Ivo P ipletikosic at gmail.com
Tue Feb 22 17:06:39 CST 2005


I've found it very useful, at least to eliminate google and similar
indexers from needlessly travesing uninteresting pages.

It also allows me to keep an eye on those that do not respect it and
watch out for unusual activity.

all in all, it takes so little effort that the downsides are very few.


On Tue, 22 Feb 2005 03:45:08 -0500, D Ross <dross202 at comcast.net> wrote:
> I've just recently learned about putting a robots.txt file in the
> directory/root. I'm not real familiar with it and what to do or if it's
> actually a good thing or a waste of time. What do you folks think?
> 
> I found this page that helps set it up:
> http://www.searchengineworld.com/robots/robots_tutorial.htm
> 
> Any other sites that explain differently or better?
> 
> Thanks,
> David
> --
> 
> * * Please support the community that supports you.  * *
> http://evolt.org/help_support_evolt/
> 
> For unsubscribe and other options, including the Tip Harvester
> and archives of thelist go to: http://lists.evolt.org
> Workers of the Web, evolt !
>


More information about the thelist mailing list