[thelist] robots.txt

Meredith Tupper wrote:
> 
> Hi all,
> 
> I'd like to set up a robots.txt file that will disallow
> access to php3 files.  I know how to disallow directories
> but I don't know how to disallow file types.  Does anyone
> have any suggestions or a sample robots.txt file I could
> pilfer?

Here's some good info:

http://info.webcrawler.com/mak/projects/robots/norobots.html

Alternatively use a meta tag:

<META NAME="ROBOTS" CONTENT="NOINDEX">
<META NAME="ROBOTS" CONTENT="NOFOLLOW">

But I believe php files are not indexed by many crawlers.

-- 
Anthony Baratta
President
KeyBoard Jockeys
                    South Park Speaks Version 3 is here!!!
                       http://www.baratta.com/southpark
                              Powered by Tsunami