[thelist] robots.txt Fwd: 404 recorded at WorldFolk.com

John C Bullas (soton.ac.uk relay) jcbullas at nildram.co.uk
Mon Feb 2 06:45:18 CST 2004

Q1: Does robots.txt provide a list of links to follow where they do not 
explicitly exist in the html on the site
or just allow or disallow directories or file extensions?

Q2: are there any advantages in having robots.txt when all my pages can be 
found by following hyperlinks


>Subject: 404 recorded at WorldFolk.com
>From: r175<at>easily.co.uk
>Mon Feb 2 2004 12:40:01 pm GMT
> tried to load 
>User Agent = Googlebot/2.1 (+http://www.googlebot.com/bot.html)

******************* John C Bullas **************
Homepage: http://www.soton.ac.uk/~johnb/
Blog: http://freeroller.net/page/johnbullas/Weblog
**** Eudora Mail  + McAfee Virus Scan 4.5.1 ****

More information about the thelist mailing list