[thelist] robots.txt Fwd: 404 recorded at WorldFolk.com
John C Bullas (soton.ac.uk relay)
jcbullas at nildram.co.uk
Mon Feb 2 06:45:18 CST 2004
Q1: Does robots.txt provide a list of links to follow where they do not
explicitly exist in the html on the site
or just allow or disallow directories or file extensions?
Q2: are there any advantages in having robots.txt when all my pages can be
found by following hyperlinks
FB
>Subject: 404 recorded at WorldFolk.com
>From: r175<at>easily.co.uk
>
>Mon Feb 2 2004 12:40:01 pm GMT
>
>64.68.82.176 tried to load
>www.fatblokeracing.org/worldfolk/simonfletcher/robots.txt
>
>User Agent = Googlebot/2.1 (+http://www.googlebot.com/bot.html)
******************* John C Bullas **************
Homepage: http://www.soton.ac.uk/~johnb/
Blog: http://freeroller.net/page/johnbullas/Weblog
**** Eudora Mail 6.0.0.22 + McAfee Virus Scan 4.5.1 ****
More information about the thelist
mailing list