What you can't recreate is the data. A company can go out of business if they can't recreate their data. Protecting the data is the most important job of any IT department. Without it, you can't pay your employees, customers, or vendors. You can't file taxes. You can't keep track of manufacturing. You can't do ANYTHING.
I am absolutely floored by the number of companies that don't take their data seriously. They don't have an intelligent backup plan at all. No thoughts on disaster recovery ever passed through their thick skull. Nothing has been tested either - it just won't happen to them, they think. Most clients I go to have a recovery failure at some point during an implementation, and it's because they have practically ignored the need to protect their data.
When I talk to clients about disaster recovery, I ask them some direct questions about the worst possible scenarios:
If this building and all its contents are destroyed (or the server room catches fire, fills with water, whatever), how much data loss will you experience?
This helps answer a number of other questions, including:
- Have they even considered this possibility?
- Are they doing any kind of off-site replication?
- If not, how often are backups moved off-site (if at all)?
- Do they store their backups in the server room? (happens more often than you think)
- How long does it take to retrieve a backup?
- How often do they back up their data?
How long will it take you to purchase new servers and have the system back up and running?
This follow up question gets them thinking in more detail about recovery.
- Do they know how many new servers, workstations, etc., it will take to get the system back up and running?
- Do they know where to purchase it and how long it will take to arrive?
- Who will reinstall all the software?
- Where will the company's operations continue after a disaster?
- Who will coordinate this effort?
Once we've covered the doom-and-gloom stuff, I move to more realistic possibilities.
If you have a hardware failure or database corruption, how much data will you lose?
In other words, how much of your company's money will you waste because of your current backup strategy. Now we're getting to the area that they may have thought about. Unfortunately, virtually every company I talk to only does full backups every night with too few (or no) transaction log backups. After this you'll know:
- How often they do a full backup.
- If they do transaction/redo log backups.
- Point-in-time restoration capability.
- Their basic backup strategy.
- How much of the company's work they can afford to lose.
Have you tested a complete recovery?
I can almost guarantee the answer to that is "no".
Have you written your disaster recovery procedures down?
Again, this is almost certainly too much to expect from most companies.
Who will perform the necessary recovery if you (the IT/database guy) are not available?
Maybe the IT guy is on vacation in Bora-Bora and he can't be contacted. Maybe he died in the server room explosion that also took out your systems. Whatever happened, it's up to someone else to do the work. In a perfect world, that person would take the company's written disaster recovery instructions and go to it. Unfortunately the previous question established they didn't test it or write it down, so they're already screwed. The thought of the IT guy not being there when disaster strikes will chill their hearts.
Okay, sorry for the long-winded post, but I gotta tell you - NOBODY pays enough attention to disaster recovery. Until, that is, an actual disaster happens. But it doesn't have to be a disaster to bring your company to your knees; Simply dropping the wrong table can cause operations to come to a screeching halt. I know - that's happened to me. Don't let it happen to you. Back up your data and write down how to restore it.
I am not surprised at all about how many companies do not have a disaster recovery plan. Nope, no surprises there.
ReplyDelete