[thelist] including a querystring in a robots.txt file
Sarah Sweeney
mr.sanders at designshift.com
Fri Apr 23 14:21:44 CDT 2004
I have my custom ColdFusion error handler set up to send me an email
every time an error is thrown. Lately I've been getting one of these
notifications spawned by a page being indexed by Yahoo. There isn't
anything wrong with the page, but somewhere there must be a link to it
with some garbage in the querystring, which is causing the error. I
can't find a "remove this url" page on Yahoo, so I'd like to just add
the offending URL to the robots.txt file. My question is, will including
the querystring in the robots.txt file cause the bot to pass over the
page only with that querystring, or will it skip that page entirely? Or
would this be something that differs depending on the bot?
Sarah
More information about the thelist
mailing list