[thelist] Robots.txt
A. Erickson
amanda at gawow.com
Mon Dec 4 20:29:49 CST 2000
I can't imagine it would make a difference -- neither help nor harm.
I have a side question, too. Robots can only crawl that which is linked,
correct? So, if I have stuff that isn't linked anywhere on my site is there
any point in including that directory as a "do not search" item?
Any robots that look at the "do not search" and purposefully search it?
- amanda (the paranoid one)
> -----Original Message-----
> From: thelist-admin at lists.evolt.org
> [mailto:thelist-admin at lists.evolt.org]On Behalf Of Jay Fitzgerald
> Sent: Monday, December 04, 2000 3:17 PM
> To: thelist at lists.evolt.org
> Subject: Re: [thelist] Robots.txt
>
>
> what do you think of just having an empty robots.txt file? That is what
> I usually use, a 0 byte file with nothing in it at all. It eliminates
> the 404 error but are there any downsides?
>
> --
> Jay Fitzgerald - N at ta$ - Internet Director
> ===================================
> Digital Athlete Gamers League http://www.dagl.net
> ===================================
> ICQ: 38823829
>
>
>
> ---------------------------------------
> For unsubscribe and other options, including
> the Tip Harvester and archive of TheList go to:
> http://lists.evolt.org Workers of the Web, evolt !
>
More information about the thelist
mailing list