[thelist] Robots.txt

[M. Hannan]

It means you had a visit from a crawler.  It looks for the existence of the 
very .txt file you mentioned...robots.txt.  Robots.txt can be used to 
encourage or exclude files from being indexed.

It is not failproof, there are some crawlers that ignore robots.txt but they 
are, by far, the minority.  Most reputable search engines abide by this.

m@


>From: "Adrian Fischer" <adrian at logo-logic.com>
>Reply-To: thelist at lists.evolt.org
>To: "The List " <thelist at lists.evolt.org>
>Subject: [thelist] Robots.txt
>Date: Tue, 5 Dec 2000 08:59:24 +1000
>
>Hi Guys,
>
>I know if you want to exclude robots form your site you can include a txt
>file to do it.  I don't have such a file but found this in my error log:
>  File does not exist: /homeblah/blah.com/robots.txt
>
>Any ideas how it got there or what it means
>
>Thanks
>
>Adrian
>
>
>---------------------------------------
>For unsubscribe and other options, including
>the Tip Harvester and archive of TheList go to:
>http://lists.evolt.org Workers of the Web, evolt !

_____________________________________________________________________________________
Get more from the Web.  FREE MSN Explorer download : http://explorer.msn.com