[fwd] RE: [thelist] Crawler pages: listing the files in a site

Jerry Bacon JBacon at MAILEX.TRA.com
Tue Jun 5 14:52:35 CDT 2001

This message was held by thelist software and
is being manually forwarded by a list admin.
Please remember to send emails in plain-text
format only, or they will not reach thelist
until it is later forwarded by a list admin.

Crawler Pages:

-----Original Message-----
From: Tony Page [mailto:zamba at zamba.com]
Sent: Friday, June 01, 2001 3:54 AM
To: thelist at lists.evolt.org
Subject: [thelist] Crawler pages: listing the files in a site

Is there an easy way to construct a crawler page, ie. a webpage containg
simple links to every (static) page in a website? I've found these very
effective in facilitating search engine etc spidering in the past, but I'd
like to extend this technique to some larger websites. A scanning program
some kind would be useful, as this would make updating easier as well.
Any ideas, anybody?
Tony Page

   <websites that work>
tel: +61 2 9953 4425
fax: +61 2 9909 8534
email: ajp at zambagrafix.com

More information about the thelist mailing list