[thelist] Very Large MySQL Table
Phil Turmel
pturmel-webdev at turmel.org
Tue Feb 10 19:21:36 CST 2009
Phil Turmel wrote:
> Hi Fred,
>
> Fred Jones wrote:
>> Data is a flat text file like this:
>>
>> aaa NS 1.2.3.4
>> bbb NS 5.6.7.8
>>
>> meaning that aaa has DNS pointed to 1.2.3.4 etc.
>>
>>> They don't use relational databases - relational databases are relatively
>>> slow in the "database" world.
>
> Have you considered using other tools?
>
> May I suggest grep?
>
> First, take the zone file and strip off all but the domain name, like so:
>
> cut -d\ -f1 <zone.txt >domains.txt
>
> (note the two spaces after the "d\")
>
> Then you can grep for patterns to your heart's content:
>
> grep copperhydride <domains.txt
>
>
> I'm betting both the size of domains.txt and the grep time are better
> than any database solution by at least an order of magnitude.
>
> HTH,
>
> Phil
Thinking about this some more... This approach is likely to be
disk-limited, not cpu-limited, so I'd used a compressed domains file,
like so:
cut -d\ -f1 <zone.txt |gzip >domains.gz
zcat domains.gz |grep copperhydride
You might even have enough excess cpu to do multiple keyword searches
together (using alternate pattern regexp notation) after a single zcat:
zcat domains.gz |grep 'copperhydride\|nickelcadmium'
Please let us know how it turns out.
Phil
More information about the thelist
mailing list