A recent conversation that we had around the dinner table revolved around fire safety hazards. The last week of November happened to be the National Home Fire Safety Week in Canada and promotes fire safety in homes and residential areas. Home insurance companies would recommend having a fire extinguisher at home placed in the kitchen or in the garage – two common places where fire starts at home. And while it’s safe to think that you have a fire extinguisher at home, there is no guarantee that it will work when you desperately need it. The only way is to inspect and test it on a regular basis according to manufacturer recommendations.
I’ve spent a fair amount of time last week trying to recover a database that got corrupted due to a bad storage subsystem. SQL Server MVP and SQLSkills CEO Paul Randal (Twitter | Blog) even wrote a blog post about a weird issue that I experienced around running SELECT statements on the table that had a corrupted page. I did manage to recover as much data as I can from the corrupted tables but the goal of this blog post is not to teach you how you can do so. In fact, if you look at the major theme of this blog, it revolves primarily around doing things right the first time so you don’t have to deal with it later. My favorite quote when dealing with situations like this is:
Prevention is better than cure. - Desiderius Erasmus Share on X
What Your Backup Vendor Promises
I’ve had opportunities in the past to evaluate and use third-party backup tools – either from a SQL Server or from an enterprise-wide backup point-of-view. I have reservations about using or recommending them because I’ve seen cases where the very tools that are supposed to protect us are the ones that cause issues. These third-party backup tools have evolved and improved throughout the years. New features were introduced that dealt with storage size requirements and the evolving needs of the modern data center.
Then, there’s virtualization-based backups where the tools will take backups of your virtual machines with the promise of taking an image backup of your SQL Server database. I’ve also seen the ugly side of these types of backups where the virtual machine image gets restored successfully but at the expense of a corrupted database. In fact, a couple of months ago, I have had to deal with a database corruption issue caused by a restored virtual machine image.
Now, don’t get me wrong. I won’t exclusively vote against using these tools. This is especially true if my customers have already spent a fair amount of time and resources implementing them across their environments. And I’m not at all saying that these tools don’t work. What I’m simply saying is that these backup vendors are missing a very important message in their marketing strategies, one that I have been highlighting for many years now. And it is something that no vendor can ever address. Yes, that is a very strong claim. Let me repeat that. No vendor can ever address these today or in the future – ever.
Disaster recovery is more than just technology. You need the right people and the proper process in place. Share on X
What Technology Will Never Solve
These third-party backup tools will tell you how good their products are. They will show you how you can do it. They will demonstrate, prove, convince, etc. That’s their job. And there is nothing wrong with that. Unfortunately, it shifts the focus of the solution to the technology rather than the more important components in a disaster recovery strategy – PEOPLE and PROCESS. In my high availability and disaster recovery training classes and workshops, I put a lot of emphasis on the PPT manifesto – people, process and technology. There is a reason why people and processes come before technology.
These third-party backup tools will come up with the latest and greatest features, the best user experience, the cheapest option. But they can never define your disaster recovery requirements. They can never define who the members of your team will be. And they can never define how your disaster recovery process will look like. These are all our responsibilities, not the tools’.
The corrupted database I tried to recover last week is one example of simply using the tools for what they are and what they offer. The customer was using a third-party virtualization-based backup tool. They use it to take backups of their SQL Server virtual machine with more than a terabyte-sized mission-critical database. The tool reported that the backups were successful. Until a real disaster happened. It’s like taking that fire extinguisher out of its box and attempting to put out a fire that is consuming your kitchen stove. Too bad you realized that it isn’t working. It’s time to call 911. That’s how I got called in.
Having the Right People and the Proper Process
While technology will continue to evolve and provide options to make our lives easier, it doesn’t change the fact that we still need people and we still need to define the proper processes. I cannot overemphasize the need to define recovery objectives and service level agreements as starting points for defining a high availability and disaster recovery strategies. They should be the basis for choosing the right solution to implement. Once those are defined, have a proper process in place defining how your disaster recovery strategies will look like. Don’t just rely on what your backup tools can provide. Evaluate how these tools can fit in your overall process. Include regular testing as part of that process. Assign the right people to constantly evaluate whether your process will meet your recovery objectives and your service level agreements.
Because the real case against SQL Server VSS- or virtualization-based backups is not the tools themselves. It is us – the people – and the processes that we define.
Additional References
- How It Works: SQL Server – VDI (VSS) Backup Resources
- Windows Server Backup May Fail Because of the SQL VSS Writer
Schedule a Call
