[thelist] My first Robot

Hershel Robinson hershelr at netvision.net.il
Tue Jul 23 06:38:01 CDT 2002

I have accepted recently to build a web robot which will search for certain
data on certain specific websites.  The client estimates that the maximal
search will consist of on the order of 1800 searches apiece on 13 different
websites.  The web sites are databases and are intended to be searched by
actual web surfers.  My concern, however, is that most human surfers do not
perform 1800 searches each separated by a few milliseconds. :)

I asked the client about asking the 13 sites if they would mind but he is
uncertain because if they do mind then what do we do?

He had an idea to rotate through the 13 sites, performing 5 or 10 searches
on each at a time.  This would spread out the searches a bit.  I still don't
think this will look like a human user however.

Our latest idea is to break the search into pieces and do a section at a
time, separating the sections by a significant length of time, like at least
an hour.

Question 1:
Does anyone have any thoughts on this?  Do I need to be concerned that the
sites will notice that they are being searched by a robot?  Will they mind?
They are not the sort of sites that would be popular among robots I don't
think--they are somewhat industry-specific.

Question 2:
Does anyone know of a *reliable* host who actually has *good* customer
support to host such a site?  The client thinks he wants a Windows platform
and we estimate that the traffic will be up to 5 Gigabytes a month, but the
bulk of that will be download not upload.

<tip type="New Clients" author="Hershel Robinson">
When dealing with a new client in a freelance relationship, always ask for
some form of payment up front.  It doesn't have to be before you actually
begin to work and it doesn't have to be a huge amount, but it is important
to get something paid early on.  This insures the seriousness of and fiscal
health of your new client before you waste a month (or two) of work.


PS: For those of you who remember my question a week or two ago about how to
estimate a job, this is that big fish I wrote about and I hooked him for an
excellent price, thanks to advice from evolters!

More information about the thelist mailing list