Are you prepared to recover your database in the event of a
disaster? Go ahead and pull up your disaster recovery plans and let’s go over
them now. I’ll wait. Oh, you don’t have it written down? Hmmm.
You’d be very surprised how many companies don’t have a
disaster plan at all, much less a plan that is written down in some way. Most
that do have a plan are dependent on a single person to execute it. Not only
that, the person in question has the entire plan in their head.
Here’s a scenario: Your DBA has always wanted to climb Mt.
Everest, and they’re going to be gone for two weeks with no contact to society
at all. Two days after they leave you get a disk failure on your primary
storage. No problem, you think, we have redundancies. Your vendor swaps out the
disk, but something is wrong: The data has magically disappeared from the redundant
disks.
Sound far-fetched? Not really. Even if you’re hosted by
Amazon AWS, you can suddenly lose all your data with no warning or explanation.
What do you do now? It’s a disaster, time to recover.
The key part of a disaster recovery operation is a good
backup. That’s a no-brainer. How often you back up and what kinds of backups is
up to your business. How much data can your company lose? 24 hours? 2 hours?
Ten seconds? That will drive your backup strategy.
Now that you have a backup, what is your Disaster Recovery plan?
Don’t tell me it’s just “restore the backup”. What about third-party software?
Cron jobs? Config entries? All those little bits and pieces you’ve cultivated
over the years could be gone. The time to figure out what those were is not
when the C-levels are breathing over your shoulder counting the money being
lost while their systems are idle.
Practice your DR plan. The plan is no good if it can’t be
executed successfully. Maybe you’ve written down a comprehensive plan, but step
17 is a reference to somebody’s blog that was taken down long ago. You’re definitely
going to want to have it all local and all spelled out. You might want to go as
far as to print it out and store it in a bank deposit box. If your disaster is
that the building was destroyed by a tornado, the easiest part is going to be
buying new hardware. The hard part is recreating your DR plan from scratch.
Don’t forget to back up your backup scripts and DR plans. It
may not be that hard to recreate a backup plan, but for the poor schlep left
behind while you’re climbing Everest - it could be impossible.
If you don’t have a Disaster Recovery plan right now, you
can create it immediately. Open up a blank document, title it “Disaster
Recovery Plan”, then type out the sentence “I do not have a disaster recovery
plan.” Email that to your boss and file it where everyone can find it. At least
they will know you are prepared.
