That syntax disallows the spider from indexing anything under http://www.yourdomain.com/source or /images or any of the other directories defined. -Chris > -----Original Message----- > From: thelist-admin at lists.evolt.org > [mailto:thelist-admin at lists.evolt.org]On Behalf Of Sean Stephens > Sent: Tuesday, November 27, 2001 1:34 PM > To: thelist at lists.evolt.org > Subject: Re: [thelist] Stopping robots.txt being read > > > > Does anyone have a link to exclusion syntax for robots? > > http://www.robotstxt.org is great but it doesn't actually give much syntax > or anything. Like, what does this mean? Does this work for all > images or is > this a folder solution? > > User-agent: * > # directed to all spiders, not just Scooter > Disallow: /source > Disallow:/photo > Disallow:/images > Disallow:/backontrack > Disallow: /cgi-bin