[thelist] robots.txt... logs files?

John C Bullas jcbullas at nildram.co.uk
Tue Nov 18 01:43:40 CST 2003

At 01:55 18/11/2003, Diane Soini wrote
>>Re: [thelist] robots.txt
>Thanks for the info. I don't have as much access to the log files as 
>probably most of you because I don't host the site myself. I suspect the 
>hosting company I use filters out some of these problems out of their own 

If you have a "nice" host you will find the logs UP from your web/http 
directory when you log in to FTP

ie. if you publish in /web click on the "parent" and look for a directory 
in root called /logs

they are certainly there for users of Easily.co.uk's hosting!!!!


>I'm mostly concerned with email addresses being stolen. I appreciate the 
>information for how to detect if that is happening. I will try that

"Mung/hide" them with fred<at>wherever.com or use  javascript to avoid 
"intact" addresses being visible when the page is read as text and not 
"rendered" by the browser....

>I'm curious if these robots have the ability to browse cgi scripts and 
>read the script without executing it. My hosting company has it set up 
>that I must put cgi scripts into a directory of my site and not cgi-bin. 
>They have likely got some protective things in place around this practice, 
>but I'm still curious.
>Spam isn't a huge problem for me, but it's the principle of it.

More information about the thelist mailing list