[thelist] Robots.txt

A. Erickson amanda at gawow.com
Tue Dec 5 11:05:00 CST 2000


That is really interesting! I know that you can specify specific bots in
terms of not indexing but it would be difficult for the average user to
figure out which bots were indexing and how.

You should write an article on thesite!

- amanda

> -----Original Message-----
> From: thelist-admin at lists.evolt.org
> [mailto:thelist-admin at lists.evolt.org]On Behalf Of Michael Buffington
> Sent: Tuesday, December 05, 2000 8:57 AM
> To: 'thelist at lists.evolt.org'
> Subject: RE: [thelist] Robots.txt
>
>
> We've hired a few folks over time that have worked directly or indirectly
> with some of Price.com's competitors.
>
> It's pretty well known that a handful of these competitors DO look in the
> "don't spider" directories.
>
> We've actually watched spiders as they've come in and watched
> them read our
> robots.txt, and either immediately step into the directory or come back to
> the directory at a later time.
>
> While I doubt anyone here is using robots.txt as a security system, I
> doesn't hurt to reiterate that it is in no way a security system.
>
> It should also be said again that most of the more well known and
> reputable
> organizations do follow the rules.  It seems that only a handful of small
> shops tend to either ignore them, are unaware of them, or choose to break
> them.
>
> Michael Buffington
> mike at price.com
> (714) 556-3890 x222
> http://www.michaelbuffington.com
> http://www.price.com
>
> -----Original Message-----
> From: A. Erickson [mailto:amanda at gawow.com]
> Sent: Monday, December 04, 2000 6:35 PM
> To: thelist at lists.evolt.org
> Subject: RE: [thelist] Robots.txt
>
>
> I can't imagine it would make a difference -- neither help nor harm.
>
> I have a side question, too. Robots can only crawl that which is linked,
> correct? So, if I have stuff that isn't linked anywhere on my
> site is there
> any point in including that directory as a "do not search" item?
>
> Any robots that look at the "do not search" and purposefully search it?
>
> - amanda (the paranoid one)
>
> > -----Original Message-----
> > From: thelist-admin at lists.evolt.org
> > [mailto:thelist-admin at lists.evolt.org]On Behalf Of Jay Fitzgerald
> > Sent: Monday, December 04, 2000 3:17 PM
> > To: thelist at lists.evolt.org
> > Subject: Re: [thelist] Robots.txt
> >
> >
> > what do you think of just having an empty robots.txt file? That is what
> > I usually use, a 0 byte file with nothing in it at all. It eliminates
> > the 404 error but are there any downsides?
> >
> > --
> > Jay Fitzgerald - N at ta$ - Internet Director
> > ===================================
> > Digital Athlete Gamers League   http://www.dagl.net
> > ===================================
> > ICQ: 38823829
> >
> >
> >
> > ---------------------------------------
> > For unsubscribe and other options, including
> > the Tip Harvester and archive of TheList go to:
> > http://lists.evolt.org Workers of the Web, evolt !
> >
>
>
> ---------------------------------------
> For unsubscribe and other options, including
> the Tip Harvester and archive of TheList go to:
> http://lists.evolt.org Workers of the Web, evolt !
>
> ---------------------------------------
> For unsubscribe and other options, including
> the Tip Harvester and archive of TheList go to:
> http://lists.evolt.org Workers of the Web, evolt !
>





More information about the thelist mailing list