[thelist] Very Large MySQL Table
pturmel-webdev at turmel.org
Tue Feb 10 19:21:36 CST 2009
Phil Turmel wrote:
> Hi Fred,
> Fred Jones wrote:
>> Data is a flat text file like this:
>> aaa NS 22.214.171.124
>> bbb NS 126.96.36.199
>> meaning that aaa has DNS pointed to 188.8.131.52 etc.
>>> They don't use relational databases - relational databases are relatively
>>> slow in the "database" world.
> Have you considered using other tools?
> May I suggest grep?
> First, take the zone file and strip off all but the domain name, like so:
> cut -d\ -f1 <zone.txt >domains.txt
> (note the two spaces after the "d\")
> Then you can grep for patterns to your heart's content:
> grep copperhydride <domains.txt
> I'm betting both the size of domains.txt and the grep time are better
> than any database solution by at least an order of magnitude.
Thinking about this some more... This approach is likely to be
disk-limited, not cpu-limited, so I'd used a compressed domains file,
cut -d\ -f1 <zone.txt |gzip >domains.gz
zcat domains.gz |grep copperhydride
You might even have enough excess cpu to do multiple keyword searches
together (using alternate pattern regexp notation) after a single zcat:
zcat domains.gz |grep 'copperhydride\|nickelcadmium'
Please let us know how it turns out.
More information about the thelist