[thesite] recently on thelist

[Daniel J. Cody]

Seth Bienek wrote:


>> would it be possible to walk through like 100 msgs at a time rather than
> 
>> try to parse the whole thing at once?

> Yes, it's possible.  The only drawback I see is that we'd have to save the
> 
> loop count in an application variable. (You're talking about running it
> 
> every x minutes, right?)
 
Actually, just for the first time when we do the initial load.. After 
that, we can do it every x minutes and won't need a loop cus it will be 
a new file(if i'm reading your last paragraph correctly)

 
> Yes, big time.  I was processing at a rate of about 8-10 messages/second on
> 
> my end.. When I copied the files up there and ran a couple of tests using
> 
> the database, it was running about 10 seconds PER message (basically taking
> 
> 100x longer).  That's one reason I was thinking that parsing it into XML and
> 
> then tossing that into the DB might be better..  I dunno though, might work

Ok, i'll see where i can get tonite on that..

Sound good?

.djc.