[thelist] Robots.txt

[Jacob Stetser]

<tip type="searchbots and blatant self-promotion">
http://littleblueeasy.com/robots/robots.php

Since last year I've been tracking which bots hit my robots.txt file 
(simple, just replaced my robots.txt file with a wee little scriptie) 
and have a significant amount of data collected. I'm thinking I might 
see if I can graph some of the information I've got tracked.

Anyone interested in any particular angle?

</tip>

>That is really interesting! I know that you can specify specific bots in
>terms of not indexing but it would be difficult for the average user to
>figure out which bots were indexing and how.
>
>You should write an article on thesite!
>
>- amanda
>
>>  -----Original Message-----
>>  From: thelist-admin at lists.evolt.org
>>  [mailto:thelist-admin at lists.evolt.org]On Behalf Of Michael Buffington
>>  Sent: Tuesday, December 05, 2000 8:57 AM
>>  To: 'thelist at lists.evolt.org'
>>  Subject: RE: [thelist] Robots.txt
>>
>>
>>  We've hired a few folks over time that have worked directly or indirectly
>>  with some of Price.com's competitors.
>>
>>  It's pretty well known that a handful of these competitors DO look in the
>>  "don't spider" directories.
>>
>>  We've actually watched spiders as they've come in and watched
>>  them read our
>>  robots.txt, and either immediately step into the directory or come back to
>>  the directory at a later time.
>>
>>  While I doubt anyone here is using robots.txt as a security system, I
>>  doesn't hurt to reiterate that it is in no way a security system.
>>
>>  It should also be said again that most of the more well known and
>>  reputable
>>  organizations do follow the rules.  It seems that only a handful of small
>>  shops tend to either ignore them, are unaware of them, or choose to break
>>  them.
>>
>>  Michael Buffington
>>  mike at price.com
>>  (714) 556-3890 x222
>>  http://www.michaelbuffington.com
>>  http://www.price.com
>>
>>  -----Original Message-----
>>  From: A. Erickson [mailto:amanda at gawow.com]
>>  Sent: Monday, December 04, 2000 6:35 PM
>>  To: thelist at lists.evolt.org
>>  Subject: RE: [thelist] Robots.txt
>>
>>
>>  I can't imagine it would make a difference -- neither help nor harm.
>>
>>  I have a side question, too. Robots can only crawl that which is linked,
>>  correct? So, if I have stuff that isn't linked anywhere on my
>>  site is there
>>  any point in including that directory as a "do not search" item?
>>
>>  Any robots that look at the "do not search" and purposefully search it?
>>
>>  - amanda (the paranoid one)
>>
>>  > -----Original Message-----
>>  > From: thelist-admin at lists.evolt.org
>>  > [mailto:thelist-admin at lists.evolt.org]On Behalf Of Jay Fitzgerald
>>  > Sent: Monday, December 04, 2000 3:17 PM
>>  > To: thelist at lists.evolt.org
>>  > Subject: Re: [thelist] Robots.txt
>>  >
>>  >
>>  > what do you think of just having an empty robots.txt file? That is what
>>  > I usually use, a 0 byte file with nothing in it at all. It eliminates
>>  > the 404 error but are there any downsides?
>>  >
>>  > --
>>  > Jay Fitzgerald - N at ta$ - Internet Director
>>  > ===================================
>>  > Digital Athlete Gamers League   http://www.dagl.net
>>  > ===================================
>>  > ICQ: 38823829
>>  >
>>  >
>>  >
>>  > ---------------------------------------
>>  > For unsubscribe and other options, including
>>  > the Tip Harvester and archive of TheList go to:
>>  > http://lists.evolt.org Workers of the Web, evolt !
>>  >
>>
>>
>>  ---------------------------------------
>>  For unsubscribe and other options, including
>>  the Tip Harvester and archive of TheList go to:
>>  http://lists.evolt.org Workers of the Web, evolt !
>>
>>  ---------------------------------------
>>  For unsubscribe and other options, including
>>  the Tip Harvester and archive of TheList go to:
>>  http://lists.evolt.org Workers of the Web, evolt !
>>
>
>
>---------------------------------------
>For unsubscribe and other options, including
>the Tip Harvester and archive of TheList go to:
>http://lists.evolt.org Workers of the Web, evolt !

-- 
icongarden.com
Making good ideas grow || http://icongarden.com/