[thelist] Queue processing with a database table.

[Bill Moseley]

On Sun, Aug 29, 2010 at 8:25 AM, Hassan Schroeder <
hassan.schroeder at gmail.com> wrote:

> On Sun, Aug 29, 2010 at 6:04 AM, Bill Moseley <moseley at hank.org> wrote:
> > I have a table called "queue" that has three columns "filename", "state",
> > and "start_time".  Each file in the table must be processed and the state
> > can be "ready", "processing", or "complete".
>
> > What I want is a way for each separate machine (well, each process) to
> grab
> > 10 rows, set them as pending and know that no other process can grab the
> > same rows.
>
> Have you considered either
>
> 1) using a daemon/background singleton process to funnel requests
>    for jobs, so there's never contention, or
>

> 2) using something fast/in-memory instead of the DB, e.g. memcached
>    or MongoDB?
>

Yes.  I was asking about using the db because that's where we already have
the data and other applications need to access it concurrently -- and why it
would be nice to avoid the table lock.  And really more curious if there was
an approach with the DB I had not considered.

Indeed, a real queue would be good.  Memcaced is possible, but it's not
persistent -- unless you were implying to use it to manage locking across
machines.  Don't have MongoDB setup -- I wish we did.  I'm sure there's a
bit of non-relational data in the RDMS that would be better in MongoDB.  It
would likely solve a lot of problems we already solved, but better...




>
>    The example on <
> http://www.mongodb.org/display/DOCS/findandmodify+Command>
>     is just what you're doing -- a process queue  :-)
>
> FWIW,
> --
> Hassan Schroeder ------------------------ hassan.schroeder at gmail.com
> twitter: @hassan
> --
>
> * * Please support the community that supports you.  * *
> http://evolt.org/help_support_evolt/
>
> For unsubscribe and other options, including the Tip Harvester
> and archives of thelist go to: http://lists.evolt.org
> Workers of the Web, evolt !
>



-- 
Bill Moseley
moseley at hank.org