[Sysadmin] Most recent outage response

[William Anderson]

On Thu, Sep 6, 2012 at 8:47 PM, Dean Mah <dean.mah at gmail.com> wrote:
> Employee Response - 2012-Sep-06 12:36 (GMT-0700) [Update 4]

cheers dean.

for some reason, dns was badly borked for some reason, i've twiddled
some knobs and levers and fixed the secondaries so that at least we
have public resolution available for 2 out of 4 nameservers in whois
and glue.  Looks like udp/53 inbound is blocked for 67.19.208.11, but
not for 67.19.208.10.  Not sure if that's an upstream network issue or
something is borked on tron (I've explicitly opened up the bind config
to listen on all IPs, but to notify and transfer only over .10, which
I've now reconfigured my secondaries to request from).

I'm going to fold tron into my rsnapshot backup regime here at home to
grab system configs and data, hopefully it'll stay up long enough for
me to pull key bits down.  In the mean time we should be seriously
considering moving, as I think I've mentioned in the past.  A single
AWS instance might do us if we have the budget for it (a t1.micro
instance costs about $10-15 a month if there's not much data stored on
it and it does nothing much CPU-wise for the month, although it's free
to "new" customers for a year).

-n