[thelist] MySQL: importing non-duplicate records

John Hicks johnlist at gulfbridge.net
Thu May 11 10:31:16 CDT 2006


Canfield, Joel wrote:
> our call detail database for the phone system has a programmatic limit
> of 100K records on the cdr table. our net admin is working on automating
> the export process to give me a dump of the data every day.
> 
> what's my best method for importing just the new records? i'm thinking
> up all kinds of complex schemes. seems like mysql has this ability built
> in, but my memory's failing me.

A few thoughts:

--Do one more sanity check with the phone system to make sure you can't 
get just the new records (e.g. via some sort of a 'tee' directive.)

--Consider lowering the size of the phone system's database to a more 
manageable size.

--Consider rotating out the phone system's cdr log every night, i.e. 
produce a completely new file every night.

--Import the phone data into a new table every night, delete as much of 
the data as possible without risking deleting the new stuff, then import 
into your  master table using the "REPLACE" command in MySQL.

Hope there's something worthwhile there.

--John



More information about the thelist mailing list