[thelist] Robots.txt
M. Hannan
desmoljones at hotmail.com
Mon Dec 4 17:04:57 CST 2000
It means you had a visit from a crawler. It looks for the existence of the
very .txt file you mentioned...robots.txt. Robots.txt can be used to
encourage or exclude files from being indexed.
It is not failproof, there are some crawlers that ignore robots.txt but they
are, by far, the minority. Most reputable search engines abide by this.
m@
>From: "Adrian Fischer" <adrian at logo-logic.com>
>Reply-To: thelist at lists.evolt.org
>To: "The List " <thelist at lists.evolt.org>
>Subject: [thelist] Robots.txt
>Date: Tue, 5 Dec 2000 08:59:24 +1000
>
>Hi Guys,
>
>I know if you want to exclude robots form your site you can include a txt
>file to do it. I don't have such a file but found this in my error log:
> File does not exist: /homeblah/blah.com/robots.txt
>
>Any ideas how it got there or what it means
>
>Thanks
>
>Adrian
>
>
>---------------------------------------
>For unsubscribe and other options, including
>the Tip Harvester and archive of TheList go to:
>http://lists.evolt.org Workers of the Web, evolt !
_____________________________________________________________________________________
Get more from the Web. FREE MSN Explorer download : http://explorer.msn.com
More information about the thelist
mailing list