[thelist] 404s during site spidering

Simon Coggins ppxsjc1 at nottingham.ac.uk
Tue Oct 23 04:52:12 CDT 2001

Hi everyone,

I am using a custom 404 page on one of my sites to send me email
notification of any broken links within my site. It works great, except
occasionally I get hit by large numbers of 404s which I'm guessing are
being generated by a spider of some kind.

The 404 page logs the referrer and the url requested, and I get around 50
messages requesting urls such as the following:

/pluto/heading.class	referrer: /pluto/
/pluto/wide.class	referrer: /pluto/
/venus/sidebar.class	referrer: /venus/
/venus/links.class	referrer: /venus/

The words heading, wide, sidebar, etc. are all classes defined in my
stylesheet. I'm not at all sure why these should be interpreted as 404s or
what I can do about it. Could I put something in a robots.txt file to stop
this from happening?

The site is at:


Any suggestions greatly appreciated.



More information about the thelist mailing list