[thelist] Robots.txt

[A. Erickson]

I can't imagine it would make a difference -- neither help nor harm.

I have a side question, too. Robots can only crawl that which is linked,
correct? So, if I have stuff that isn't linked anywhere on my site is there
any point in including that directory as a "do not search" item?

Any robots that look at the "do not search" and purposefully search it?

- amanda (the paranoid one)

> -----Original Message-----
> From: thelist-admin at lists.evolt.org
> [mailto:thelist-admin at lists.evolt.org]On Behalf Of Jay Fitzgerald
> Sent: Monday, December 04, 2000 3:17 PM
> To: thelist at lists.evolt.org
> Subject: Re: [thelist] Robots.txt
>
>
> what do you think of just having an empty robots.txt file? That is what
> I usually use, a 0 byte file with nothing in it at all. It eliminates
> the 404 error but are there any downsides?
>
> --
> Jay Fitzgerald - N at ta$ - Internet Director
> ===================================
> Digital Athlete Gamers League   http://www.dagl.net
> ===================================
> ICQ: 38823829
>
>
>
> ---------------------------------------
> For unsubscribe and other options, including
> the Tip Harvester and archive of TheList go to:
> http://lists.evolt.org Workers of the Web, evolt !
>