[thelist] Robots.txt

Seb Barre sebastien at oven.com
Tue Dec 5 09:32:43 CST 2000

At 06:35 PM 12/4/2000 -0800, you wrote:
>I can't imagine it would make a difference -- neither help nor harm.
>I have a side question, too. Robots can only crawl that which is linked,
>correct? So, if I have stuff that isn't linked anywhere on my site is there
>any point in including that directory as a "do not search" item?

Yes, robots follow links. If it's not linked, it won't get 
indexed.  Although remember that web directories that are linked but have 
no index files will display a directory contents (by default at 
least.  Your webserver can be configured otherwise), and those files are 
likely to be indexed.  But if you have a document in a directory (with an 
index file or Indexing turned off) that isn't linked anywhere, it won't be 
indexed because the crawler has no way of reaching it.

Crawlers act just like web browsers and are subject to the same rules and 
restrictions a webmaster imposes on human surfers, plus whatever is defined 
in your robots.txt file.

--- -- -
Seb Barre - seb at oven.com
OVEN Digital Toronto
Work: 416-595-9750 x 222
Mobile: 416-254-5078

More information about the thelist mailing list