[thelist] MySQL: importing non-duplicate records
John Hicks
johnlist at gulfbridge.net
Thu May 11 10:31:16 CDT 2006
Canfield, Joel wrote:
> our call detail database for the phone system has a programmatic limit
> of 100K records on the cdr table. our net admin is working on automating
> the export process to give me a dump of the data every day.
>
> what's my best method for importing just the new records? i'm thinking
> up all kinds of complex schemes. seems like mysql has this ability built
> in, but my memory's failing me.
A few thoughts:
--Do one more sanity check with the phone system to make sure you can't
get just the new records (e.g. via some sort of a 'tee' directive.)
--Consider lowering the size of the phone system's database to a more
manageable size.
--Consider rotating out the phone system's cdr log every night, i.e.
produce a completely new file every night.
--Import the phone data into a new table every night, delete as much of
the data as possible without risking deleting the new stuff, then import
into your master table using the "REPLACE" command in MySQL.
Hope there's something worthwhile there.
--John
More information about the thelist
mailing list