[thelist] Queue processing with a database table.
Bill Moseley
moseley at hank.org
Sun Aug 29 12:28:17 CDT 2010
On Sun, Aug 29, 2010 at 8:25 AM, Hassan Schroeder <
hassan.schroeder at gmail.com> wrote:
> On Sun, Aug 29, 2010 at 6:04 AM, Bill Moseley <moseley at hank.org> wrote:
> > I have a table called "queue" that has three columns "filename", "state",
> > and "start_time". Each file in the table must be processed and the state
> > can be "ready", "processing", or "complete".
>
> > What I want is a way for each separate machine (well, each process) to
> grab
> > 10 rows, set them as pending and know that no other process can grab the
> > same rows.
>
> Have you considered either
>
> 1) using a daemon/background singleton process to funnel requests
> for jobs, so there's never contention, or
>
> 2) using something fast/in-memory instead of the DB, e.g. memcached
> or MongoDB?
>
Yes. I was asking about using the db because that's where we already have
the data and other applications need to access it concurrently -- and why it
would be nice to avoid the table lock. And really more curious if there was
an approach with the DB I had not considered.
Indeed, a real queue would be good. Memcaced is possible, but it's not
persistent -- unless you were implying to use it to manage locking across
machines. Don't have MongoDB setup -- I wish we did. I'm sure there's a
bit of non-relational data in the RDMS that would be better in MongoDB. It
would likely solve a lot of problems we already solved, but better...
>
> The example on <
> http://www.mongodb.org/display/DOCS/findandmodify+Command>
> is just what you're doing -- a process queue :-)
>
> FWIW,
> --
> Hassan Schroeder ------------------------ hassan.schroeder at gmail.com
> twitter: @hassan
> --
>
> * * Please support the community that supports you. * *
> http://evolt.org/help_support_evolt/
>
> For unsubscribe and other options, including the Tip Harvester
> and archives of thelist go to: http://lists.evolt.org
> Workers of the Web, evolt !
>
--
Bill Moseley
moseley at hank.org
More information about the thelist
mailing list