[thelist] Server Script Reading HTML

[David Kaufman]

Chris <axe at suburbia.com.au> queried thusly:
>
> I have to code some server side scripts which have a requirement to
> read some web pages via HTTP and take action depending on what it
> finds. I am thinking perl would be the best way to go however I have
> not done anything like this with perl before, in fact I've only done
> this before with VB. The tricky part is that the script I code must
> be able to give a certain user agent aswell as a specific referer.
>
> Can anyone point me in the right direction for help / tutorials for
> this kind of thing.

*pointing_to-> perl's most excellent LWP module:
http://search.cpan.org/author/GAAS/libwww-perl-5.65/lib/LWP.pm

and the relevent perl.com FAQ:
http://www.perldoc.com/perl5.6/pod/perlfaq9.html#How-do-I-fetch-an-HTML-file
-

  # simplest solution

  use LWP::UserAgent;
  my $ua = LWP::UserAgent->new;
  my $ua->agent('Mozilla/5.0');
  my $request = HTTP::Request->new(
    'GET',
    'http://www.url.org/whutevuh'
  );
  my $response = $ua->request($request);

  if ($response->is_success) {
    print $response->content;
  } else {
    print "Bad news: " . $response->message;
  }

 # fin

now that wasn't so hard, was it ? :-)

LWP::Simple is even simpler, of course :-)  while less simple interfaces to
LWP allow making POST requests, submitting Basic-Auth usernames and
passwords, accepting and returning cookies, making SSL requests, doing HTTP
uploads, converting the retrieved HTML into plaintext, and following
30X-Moved style redirection response headers.  but perl does try to keep
simple things simple, and LWP is a good example of the philosophy in
practice

hth,

-dave

--
sig {
  # stripped by Morton AntiVirus SMTP filter ...it's
  # for your own protection -- you'll thank us later.
}