[thelist] robots.txt

Diane Soini dianesoini at earthlink.net
Mon Nov 17 19:55:53 CST 2003

> Re: [thelist] robots.txt

Thanks for the info. I don't have as much access to the log files as 
probably most of you because I don't host the site myself. I suspect 
the hosting company I use filters out some of these problems out of 
their own self-interest.

I'm mostly concerned with email addresses being stolen. I appreciate 
the information for how to detect if that is happening. I will try that.

I'm curious if these robots have the ability to browse cgi scripts and 
read the script without executing it. My hosting company has it set up 
that I must put cgi scripts into a directory of my site and not 
cgi-bin. They have likely got some protective things in place around 
this practice, but I'm still curious.

Spam isn't a huge problem for me, but it's the principle of it.

More information about the thelist mailing list