[thelist] Robots.txt, the robots meta tag, and copyright references needed.
April
april at farstrider.org
Mon Jan 14 16:23:42 CST 2002
Hey all,
I am in the middle of the most ridiculous battle with our main technical
department in the history of man. And I thought the one where they put
special characters in the database passwords, because they thought it would
be more secure if we couldn't type in the password and had to copy and
paste it from an email from them was bad...
Here is their chain of thought, where they got it I don't know:
1. Having a robots.txt prevents people from copying our information on our
websites.
2. If we don't have a robots.txt disallowing all access, we are giving
people a legal right to take our information.
3. Besides that, the robots.txt physically prevents all web spiders from
accessing our site.
4. We should contact search engines and tell them our keywords... It
might take a bit of following up, but that's what I'm for. (Gods, I can
see that email now... Dear Google...)
5. Since I'm so difficult, they have found a way to add a NOFOLLOW robots
meta tag to the front page, so search engines can read that... no, we can't
take down the robots.txt and put robots meta tags on other pages.
I don't know how they decided that robots.txt's are a legal issue, but I
don't think I can convince them otherwise without the name of an important
person behind it. Can anyone point me to articles which -don't- refer to
robots.txt as a security measure, and explain why not? And if anyone has
ever seen anything about legal issues involving robots.txt, if such even
exist, I would really love those links. Also, I'm looking for an article
on those email harvesters which will use a robots.txt to choose where to
index first.
(yes, I'm searching on google, but most of it assumes that the person
reading it has common sense and the rest seems to agree that using a
robots.txt will protect your data).
More information about the thelist
mailing list