Got a nice mention on this post for my CHUG presentation a while back.

Since then we’ve actually ditched the linux-ha setup.  Turns out if you buy hardware that is halfway decent, the chances of failure pretty low.  If you keep your NN metadata on NFS storage you can get up and running quickly again someplace else.  Best thing to do is to use a service DNS CNAME for your NN and JT services (as things like Hive like to write the URI into the DB) so moving the service is just a DNS change.

I’m trying to come up with a followup presentation on progress (or lack thereof) on those next steps.  The working title for the presentation is “Logs are Evil” which really gets at the root of the problem with logs.  Any piece of data that your system generates needs a destination and a lifecycle (retention period, etc).  If you don’t do this early, like my kid’s messy room, soon becomes an unwieldy cleanup project.