We've hired a few folks over time that have worked directly or indirectly with some of Price.com's competitors. It's pretty well known that a handful of these competitors DO look in the "don't spider" directories. We've actually watched spiders as they've come in and watched them read our robots.txt, and either immediately step into the directory or come back to the directory at a later time. While I doubt anyone here is using robots.txt as a security system, I doesn't hurt to reiterate that it is in no way a security system. It should also be said again that most of the more well known and reputable organizations do follow the rules. It seems that only a handful of small shops tend to either ignore them, are unaware of them, or choose to break them. Michael Buffington mike at price.com (714) 556-3890 x222 http://www.michaelbuffington.com http://www.price.com -----Original Message----- From: A. Erickson [mailto:amanda at gawow.com] Sent: Monday, December 04, 2000 6:35 PM To: thelist at lists.evolt.org Subject: RE: [thelist] Robots.txt I can't imagine it would make a difference -- neither help nor harm. I have a side question, too. Robots can only crawl that which is linked, correct? So, if I have stuff that isn't linked anywhere on my site is there any point in including that directory as a "do not search" item? Any robots that look at the "do not search" and purposefully search it? - amanda (the paranoid one) > -----Original Message----- > From: thelist-admin at lists.evolt.org > [mailto:thelist-admin at lists.evolt.org]On Behalf Of Jay Fitzgerald > Sent: Monday, December 04, 2000 3:17 PM > To: thelist at lists.evolt.org > Subject: Re: [thelist] Robots.txt > > > what do you think of just having an empty robots.txt file? That is what > I usually use, a 0 byte file with nothing in it at all. It eliminates > the 404 error but are there any downsides? > > -- > Jay Fitzgerald - N at ta$ - Internet Director > =================================== > Digital Athlete Gamers League http://www.dagl.net > =================================== > ICQ: 38823829 > > > > --------------------------------------- > For unsubscribe and other options, including > the Tip Harvester and archive of TheList go to: > http://lists.evolt.org Workers of the Web, evolt ! > --------------------------------------- For unsubscribe and other options, including the Tip Harvester and archive of TheList go to: http://lists.evolt.org Workers of the Web, evolt !