[thelist] My first Robot

[Hershel Robinson]

----- Original Message -----
From: "Hershel Robinson" <hershelr at netvision.net.il>
To: <thelist at lists.evolt.org>
Sent: Tuesday, July 23, 2002 2:44 PM
Subject: [thelist] My first Robot


> I have accepted recently to build a web robot which will search for
certain
> data on certain specific websites.  The client estimates that the maximal
> search will consist of on the order of 1800 searches apiece on 13
different
> websites.  The web sites are databases and are intended to be searched by
> actual web surfers.  My concern, however, is that most human surfers do
not
> perform 1800 searches each separated by a few milliseconds. :)
>
> I asked the client about asking the 13 sites if they would mind but he is
> uncertain because if they do mind then what do we do?
>
> He had an idea to rotate through the 13 sites, performing 5 or 10 searches
> on each at a time.  This would spread out the searches a bit.  I still
don't
> think this will look like a human user however.
>
> Our latest idea is to break the search into pieces and do a section at a
> time, separating the sections by a significant length of time, like at
least
> an hour.
>
> Question 1:
> Does anyone have any thoughts on this?  Do I need to be concerned that the
> sites will notice that they are being searched by a robot?  Will they
mind?
> They are not the sort of sites that would be popular among robots I don't
> think--they are somewhat industry-specific.
>
> Question 2:
> Does anyone know of a *reliable* host who actually has *good* customer
> support to host such a site?  The client thinks he wants a Windows
platform
> and we estimate that the traffic will be up to 5 Gigabytes a month, but
the
> bulk of that will be download not upload.
>
> <tip type="New Clients" author="Hershel Robinson">
> When dealing with a new client in a freelance relationship, always ask for
> some form of payment up front.  It doesn't have to be before you actually
> begin to work and it doesn't have to be a huge amount, but it is important
> to get something paid early on.  This insures the seriousness of and
fiscal
> health of your new client before you waste a month (or two) of work.
> </tip>
>
> Thanks,
> Hershel
>
> PS: For those of you who remember my question a week or two ago about how
to
> estimate a job, this is that big fish I wrote about and I hooked him for
an
> excellent price, thanks to advice from evolters!
>
> --
> For unsubscribe and other options, including
> the Tip Harvester and archive of thelist go to:
> http://lists.evolt.org Workers of the Web, evolt !