[thelist] Robots.txt

[Mike Smith]

I would expect that this has been caused by a robot checking for the existence of a robots.txt file (the only way it could do this
would be to request the page). It sounds like the webserver is logging every file not found error.

HTH

Mike

> -----Original Message-----
> From: thelist-admin at lists.evolt.org
> [mailto:thelist-admin at lists.evolt.org]On Behalf Of Adrian Fischer
> Sent: 04 December 2000 17:59
> To: The List
> Subject: [thelist] Robots.txt
>
>
> Hi Guys,
>
> I know if you want to exclude robots form your site you can include a txt
> file to do it.  I don't have such a file but found this in my error log:
>  File does not exist: /homeblah/blah.com/robots.txt
>
> Any ideas how it got there or what it means
>
> Thanks
>
> Adrian
>
>
> ---------------------------------------
> For unsubscribe and other options, including
> the Tip Harvester and archive of TheList go to:
> http://lists.evolt.org Workers of the Web, evolt !