[thelist] including a querystring in a robots.txt file

Sarah Sweeney mr.sanders at designshift.com
Fri Apr 23 14:21:44 CDT 2004


I have my custom ColdFusion error handler set up to send me an email 
every time an error is thrown. Lately I've been getting one of these 
notifications spawned by a page being indexed by Yahoo. There isn't 
anything wrong with the page, but somewhere there must be a link to it 
with some garbage in the querystring, which is causing the error. I 
can't find a "remove this url" page on Yahoo, so I'd like to just add 
the offending URL to the robots.txt file. My question is, will including 
the querystring in the robots.txt file cause the bot to pass over the 
page only with that querystring, or will it skip that page entirely? Or 
would this be something that differs depending on the bot?

Sarah



More information about the thelist mailing list