[thelist] robots.txt and subdomains
John C Bullas
jcbullas at nildram.co.uk
Mon Dec 8 13:11:56 CST 2003
>
>
>I want to make sure that all the files in the foobar folder are not indexed.
you need to assume anything that will try to index your site will conform
to the robots standard...
which it won't :)
....Otherwise you need to explicitly allow the right crawlers in and block
all the others?
and thre others can play at being the nice guys so it needs to be by IP?
FB
I believe in... if you don't want it found, don't put it on the web
More information about the thelist
mailing list