When was the last time you practiced recovery of your production database?

Are you prepared to recover your database in the event of a disaster? Go ahead and pull up your disaster recovery plans and let’s go over them now. I’ll wait. Oh, you don’t have it written down? Hmmm.

You’d be very surprised how many companies don’t have a disaster plan at all, much less a plan that is written down in some way. Most that do have a plan are dependent on a single person to execute it. Not only that, the person in question has the entire plan in their head.

Here’s a scenario: Your DBA has always wanted to climb Mt. Everest, and they’re going to be gone for two weeks with no contact to society at all. Two days after they leave you get a disk failure on your primary storage. No problem, you think, we have redundancies. Your vendor swaps out the disk, but something is wrong: The data has magically disappeared from the redundant disks.

Sound far-fetched? Not really. Even if you’re hosted by Amazon AWS, you can suddenly lose all your data with no warning or explanation. What do you do now? It’s a disaster, time to recover.

The key part of a disaster recovery operation is a good backup. That’s a no-brainer. How often you back up and what kinds of backups is up to your business. How much data can your company lose? 24 hours? 2 hours? Ten seconds? That will drive your backup strategy.

Now that you have a backup, what is your Disaster Recovery plan? Don’t tell me it’s just “restore the backup”. What about third-party software? Cron jobs? Config entries? All those little bits and pieces you’ve cultivated over the years could be gone. The time to figure out what those were is not when the C-levels are breathing over your shoulder counting the money being lost while their systems are idle.

Practice your DR plan. The plan is no good if it can’t be executed successfully. Maybe you’ve written down a comprehensive plan, but step 17 is a reference to somebody’s blog that was taken down long ago. You’re definitely going to want to have it all local and all spelled out. You might want to go as far as to print it out and store it in a bank deposit box. If your disaster is that the building was destroyed by a tornado, the easiest part is going to be buying new hardware. The hard part is recreating your DR plan from scratch.

Don’t forget to back up your backup scripts and DR plans. It may not be that hard to recreate a backup plan, but for the poor schlep left behind while you’re climbing Everest - it could be impossible.

If you don’t have a Disaster Recovery plan right now, you can create it immediately. Open up a blank document, title it “Disaster Recovery Plan”, then type out the sentence “I do not have a disaster recovery plan.” Email that to your boss and file it where everyone can find it. At least they will know you are prepared.