[thelist] robots.txt and subdomains

[John C Bullas]

>
>
>I want to make sure that all the files in the foobar folder are not indexed.

you need to assume anything that will try to index your site will conform 
to the robots standard...
which it won't :)

....Otherwise you need to explicitly allow the right crawlers in and block 
all the others?
and thre others can play at being the nice guys so it needs to be by IP?

FB
I believe in... if you don't want it found, don't put it on the web