[thelist] Queue processing with a database table.

Hassan Schroeder hassan.schroeder at gmail.com
Sun Aug 29 10:25:58 CDT 2010


On Sun, Aug 29, 2010 at 6:04 AM, Bill Moseley <moseley at hank.org> wrote:
> I have a table called "queue" that has three columns "filename", "state",
> and "start_time".  Each file in the table must be processed and the state
> can be "ready", "processing", or "complete".

> What I want is a way for each separate machine (well, each process) to grab
> 10 rows, set them as pending and know that no other process can grab the
> same rows.

Have you considered either

1) using a daemon/background singleton process to funnel requests
    for jobs, so there's never contention, or

2) using something fast/in-memory instead of the DB, e.g. memcached
    or MongoDB?

    The example on <http://www.mongodb.org/display/DOCS/findandmodify+Command>
     is just what you're doing -- a process queue  :-)

FWIW,
-- 
Hassan Schroeder ------------------------ hassan.schroeder at gmail.com
twitter: @hassan


More information about the thelist mailing list