[thelist] Stopping robots.txt being read

Chris Blessing webguy at mail.rit.edu
Tue Nov 27 12:39:08 CST 2001


That syntax disallows the spider from indexing anything under
http://www.yourdomain.com/source or /images or any of the other directories
defined.

-Chris

> -----Original Message-----
> From: thelist-admin at lists.evolt.org
> [mailto:thelist-admin at lists.evolt.org]On Behalf Of Sean Stephens
> Sent: Tuesday, November 27, 2001 1:34 PM
> To: thelist at lists.evolt.org
> Subject: Re: [thelist] Stopping robots.txt being read
>
>
>
> Does anyone have a link to exclusion syntax for robots?
>
> http://www.robotstxt.org is great but it doesn't actually give much syntax
> or anything. Like, what does this mean? Does this work for all
> images or is
> this a folder solution?
>
> User-agent: *
> # directed to all spiders, not just Scooter
> Disallow: /source
> Disallow:/photo
> Disallow:/images
> Disallow:/backontrack
> Disallow: /cgi-bin





More information about the thelist mailing list