[thelist] Is there a tool to follow links and report them?

[Tony Crockford]

On Wed, 25 Feb 2004 17:44:24 -0800, Diane Soini <dianesoini at earthlink.net> 
wrote:

> I know that you can download entire web sites, but what I want to do is 
> not download a web site, but collect a list of all the links. Does a 
> tool like this exist?
>
> The reason is that I need to put together a list of a bunch of 
> multimedia presentations that are linked all over inside an intranet and 
> a web site. Instead of copying and pasting and trying to search for them 
> myself, I was wondering if there was some kind of tool that would crawl 
> the web site and collect all the URLs. Even better would be if it would 
> remove duplicates and/or allow me to put in a regexp to be more 
> selective.
>
> Is there something like this out there?

This is a link collecting (and more) spin off from Htdig (the opensource 
search engine) and might do what you're looking for:

http://htcheck.sourceforge.net/

hth

Tony





-- 
listening to: "if you don't know me by now" by David Brent
http://www.xebit.net/