[thelist] Crawler pages: listing the files in a site

[Tony Page]

Is there an easy way to construct a crawler page, ie. a webpage containg
simple links to every (static) page in a website? I've found these very
effective in facilitating search engine etc spidering in the past, but I'd
like to extend this technique to some larger websites. A scanning program of
some kind would be useful, as this would make updating easier as well.
Any ideas, anybody?
Tony Page

      [ZambaGrafix]
   <websites that work>
tel: +61 2 9953 4425
fax: +61 2 9909 8534
email: ajp at zambagrafix.com
http://www.zambagrafix.com