[thelist] Cluster/Load Balancing Issues.....

Steve Lewis slewis at macrovista.net
Thu Sep 5 11:49:01 CDT 2002


Chris:

> files via http to our server.  The process works fine as is on a single box,
> but
> when we throw it into a cluster, we need to gaurantee that it either stays
> on
> that box or points to a specific box.
>
> We thought about just pointing the upload to a specific IP address for the
> main
> box, but the load balancer might end up sending the request to any of the
> boxes.

File replication across the cluster should be available, but sometimes
we have to brew our own.

This requires that you have a way to individually address each server in
the cluster.  It sounds like this might be a problem, or else you would
probably just reference the individual server in the link to the
uploaded file in the first place.  I will describe two possible
solutions briefly and hope that they give you some ideas...as they
probably will present you with new problems themselves.  Only the first
one attempts file replication.

Method A) I have seen hosts who, on upload to an ftp server will queue
the file in a process which will then move the file from the ftp server
to the clustered web servers.  This keeps everything you need for a site
together on each web server, and thus is the most congruent sort of
option to someone who hosts in a clustered environment.  Downside is
that the availability of the files someone uploads is delayed while the
batch process works.

(best viewed in fixed-point fonts)

   Public IPs   ||   Private IPs
  Cluster Hub   ||  Web Server 1
  FTP server    ||  Web Server 2
                ||  Web Server 3
                ||  Web Server 4

step
1) file uploaded to FTP server
2) batch process runs every two minutes to copy changed/new files to
each of the 4 web servers.

Method B) Get the uploaded files off the clustered servers, locate them
on a separate server with a separate hostname:  if you manage
example.com which is hosted on the cluster, put uploaded files on
uploads.example.com ... This is done by having the form submit to a url
on uploads.example.com OR by having the scripts on the clustered servers
copy the uploaded file to uploads.example.com after processing on the
web server.  You sacrifice the load balancing benefits and redundancy of
the cluster for uploaded files, and thus most of the value of the
cluster for this option.  It does provide what could be a simpler
solution, and does not impose a significant delay between the upload of
the form and the availability of the file.

(best viewed in fixed-point fonts)

   Public IPs   ||   Private IPs
  Cluster Hub   ||  Web Server 1
  upload server ||  Web Server 2
                ||  Web Server 3
                ||  Web Server 4


HTH
-- Steve




More information about the thelist mailing list