[thelist] Queue processing with a database table.
Hassan Schroeder
hassan.schroeder at gmail.com
Sun Aug 29 10:25:58 CDT 2010
On Sun, Aug 29, 2010 at 6:04 AM, Bill Moseley <moseley at hank.org> wrote:
> I have a table called "queue" that has three columns "filename", "state",
> and "start_time". Each file in the table must be processed and the state
> can be "ready", "processing", or "complete".
> What I want is a way for each separate machine (well, each process) to grab
> 10 rows, set them as pending and know that no other process can grab the
> same rows.
Have you considered either
1) using a daemon/background singleton process to funnel requests
for jobs, so there's never contention, or
2) using something fast/in-memory instead of the DB, e.g. memcached
or MongoDB?
The example on <http://www.mongodb.org/display/DOCS/findandmodify+Command>
is just what you're doing -- a process queue :-)
FWIW,
--
Hassan Schroeder ------------------------ hassan.schroeder at gmail.com
twitter: @hassan
More information about the thelist
mailing list