[thelist] robots.txt Fwd: 404 recorded at WorldFolk.com

John C Bullas (soton.ac.uk relay) jcbullas at nildram.co.uk
Mon Feb 2 06:45:18 CST 2004


Q1: Does robots.txt provide a list of links to follow where they do not 
explicitly exist in the html on the site
or just allow or disallow directories or file extensions?

Q2: are there any advantages in having robots.txt when all my pages can be 
found by following hyperlinks

FB


>Subject: 404 recorded at WorldFolk.com
>From: r175<at>easily.co.uk
>
>Mon Feb 2 2004 12:40:01 pm GMT
>
>64.68.82.176 tried to load 
>www.fatblokeracing.org/worldfolk/simonfletcher/robots.txt
>
>User Agent = Googlebot/2.1 (+http://www.googlebot.com/bot.html)

******************* John C Bullas **************
Homepage: http://www.soton.ac.uk/~johnb/
Blog: http://freeroller.net/page/johnbullas/Weblog
**** Eudora Mail 6.0.0.22  + McAfee Virus Scan 4.5.1 ****



More information about the thelist mailing list