[thelist] Robots.txt
Seb Barre
sebastien at oven.com
Tue Dec 5 09:32:43 CST 2000
At 06:35 PM 12/4/2000 -0800, you wrote:
>I can't imagine it would make a difference -- neither help nor harm.
>
>I have a side question, too. Robots can only crawl that which is linked,
>correct? So, if I have stuff that isn't linked anywhere on my site is there
>any point in including that directory as a "do not search" item?
Yes, robots follow links. If it's not linked, it won't get
indexed. Although remember that web directories that are linked but have
no index files will display a directory contents (by default at
least. Your webserver can be configured otherwise), and those files are
likely to be indexed. But if you have a document in a directory (with an
index file or Indexing turned off) that isn't linked anywhere, it won't be
indexed because the crawler has no way of reaching it.
Crawlers act just like web browsers and are subject to the same rules and
restrictions a webmaster imposes on human surfers, plus whatever is defined
in your robots.txt file.
--- -- -
Seb Barre - seb at oven.com
OVEN Digital Toronto
Work: 416-595-9750 x 222
Mobile: 416-254-5078
http://www.oven.com/
More information about the thelist
mailing list