[thelist] Robots.txt

Michael Buffington mike at price.com
Tue Dec 5 10:58:41 CST 2000


We've hired a few folks over time that have worked directly or indirectly
with some of Price.com's competitors.

It's pretty well known that a handful of these competitors DO look in the
"don't spider" directories.

We've actually watched spiders as they've come in and watched them read our
robots.txt, and either immediately step into the directory or come back to
the directory at a later time.

While I doubt anyone here is using robots.txt as a security system, I
doesn't hurt to reiterate that it is in no way a security system.

It should also be said again that most of the more well known and reputable
organizations do follow the rules.  It seems that only a handful of small
shops tend to either ignore them, are unaware of them, or choose to break
them.

Michael Buffington
mike at price.com
(714) 556-3890 x222
http://www.michaelbuffington.com
http://www.price.com 

-----Original Message-----
From: A. Erickson [mailto:amanda at gawow.com]
Sent: Monday, December 04, 2000 6:35 PM
To: thelist at lists.evolt.org
Subject: RE: [thelist] Robots.txt


I can't imagine it would make a difference -- neither help nor harm.

I have a side question, too. Robots can only crawl that which is linked,
correct? So, if I have stuff that isn't linked anywhere on my site is there
any point in including that directory as a "do not search" item?

Any robots that look at the "do not search" and purposefully search it?

- amanda (the paranoid one)

> -----Original Message-----
> From: thelist-admin at lists.evolt.org
> [mailto:thelist-admin at lists.evolt.org]On Behalf Of Jay Fitzgerald
> Sent: Monday, December 04, 2000 3:17 PM
> To: thelist at lists.evolt.org
> Subject: Re: [thelist] Robots.txt
>
>
> what do you think of just having an empty robots.txt file? That is what
> I usually use, a 0 byte file with nothing in it at all. It eliminates
> the 404 error but are there any downsides?
>
> --
> Jay Fitzgerald - N at ta$ - Internet Director
> ===================================
> Digital Athlete Gamers League   http://www.dagl.net
> ===================================
> ICQ: 38823829
>
>
>
> ---------------------------------------
> For unsubscribe and other options, including
> the Tip Harvester and archive of TheList go to:
> http://lists.evolt.org Workers of the Web, evolt !
>


---------------------------------------
For unsubscribe and other options, including
the Tip Harvester and archive of TheList go to:
http://lists.evolt.org Workers of the Web, evolt ! 




More information about the thelist mailing list