[thelist] Robots.txt, the robots meta tag, and copyright references needed.

April april at farstrider.org
Mon Jan 14 16:23:42 CST 2002


Hey all,

I am in the middle of the most ridiculous battle with our main technical 
department in the history of man.  And I thought the one where they put 
special characters in the database passwords, because they thought it would 
be more secure if we couldn't type in the password and had to copy and 
paste it from an email from them was bad...

Here is their chain of thought, where they got it I don't know:
1.  Having a robots.txt prevents people from copying our information on our 
websites.
2.  If we don't have a robots.txt disallowing all access, we are giving 
people a legal right to take our information.
3.  Besides that, the robots.txt physically prevents all web spiders from 
accessing our site.
4.  We should contact search engines and tell them our keywords...  It 
might take a bit of following up, but that's what I'm for.  (Gods, I can 
see that email now... Dear Google...)
5.   Since I'm so difficult, they have found a way to add a NOFOLLOW robots 
meta tag to the front page, so search engines can read that... no, we can't 
take down the robots.txt and put robots meta tags on other pages.

I don't know how they decided that robots.txt's are a legal issue, but I 
don't think I can convince them otherwise without the name of an important 
person behind it.  Can anyone point me to articles which -don't- refer to 
robots.txt as a security measure, and explain why not?  And if anyone has 
ever seen anything about legal issues involving robots.txt, if such even 
exist, I would really love those links.  Also, I'm looking for an article 
on those email harvesters which will use a robots.txt to choose where to 
index first.

(yes, I'm searching on google, but most of it assumes that the person 
reading it has common sense and the rest seems to agree that using a 
robots.txt will protect your data). 




More information about the thelist mailing list