[thelist] Is there a tool to follow links and report them?

Tony Crockford tonyc at boldfish.co.uk
Thu Feb 26 01:22:04 CST 2004

On Wed, 25 Feb 2004 17:44:24 -0800, Diane Soini <dianesoini at earthlink.net> 

> I know that you can download entire web sites, but what I want to do is 
> not download a web site, but collect a list of all the links. Does a 
> tool like this exist?
> The reason is that I need to put together a list of a bunch of 
> multimedia presentations that are linked all over inside an intranet and 
> a web site. Instead of copying and pasting and trying to search for them 
> myself, I was wondering if there was some kind of tool that would crawl 
> the web site and collect all the URLs. Even better would be if it would 
> remove duplicates and/or allow me to put in a regexp to be more 
> selective.
> Is there something like this out there?

This is a link collecting (and more) spin off from Htdig (the opensource 
search engine) and might do what you're looking for:




listening to: "if you don't know me by now" by David Brent

More information about the thelist mailing list