[thelist] robots.txt Fwd: 404 recorded at WorldFolk.com

[John C Bullas]

At 13:23 02/02/2004, you wrote
> > -----Original Message-----
> > From: John C Bullas (soton.ac.uk relay)
> > Sent: Monday, February 02, 2004 7:45 AM
> >
> > Q1: Does robots.txt provide a list of links to follow where they do not
> > explicitly exist in the html on the site
> > or just allow or disallow directories or file extensions?
>
>John,
>
>No, it just explicitly SUGGESTS which folders and directories not to enter.
>Robots may or may not follow this suggestion.

I'll stick with my hidden "spiders this way" page ;)

with directory listing disabled

FB