You might want to try this.... http://www.mcanerin.com/EN/search-engine/robots-txt.asp One difference between the web-page generator and your code is that your default rule is first. That should be last, me thinks.