[thelist] 404s during site spidering

[Simon Coggins]

Hi everyone,

I am using a custom 404 page on one of my sites to send me email
notification of any broken links within my site. It works great, except
occasionally I get hit by large numbers of 404s which I'm guessing are
being generated by a spider of some kind.

The 404 page logs the referrer and the url requested, and I get around 50
messages requesting urls such as the following:

/pluto/heading.class	referrer: /pluto/
/pluto/wide.class	referrer: /pluto/
/venus/sidebar.class	referrer: /venus/
/venus/links.class	referrer: /venus/
etc.

The words heading, wide, sidebar, etc. are all classes defined in my
stylesheet. I'm not at all sure why these should be interpreted as 404s or
what I can do about it. Could I put something in a robots.txt file to stop
this from happening?

The site is at:

http://www.solarsystem.f2s.com/

Any suggestions greatly appreciated.

Thanks,

Simon