Fun Fact: This story won me a FREE SQLSkills training back in 2012.
My official job title was SQL Server database administrator.
But because I was the only one in the team with System Center experience, I instantly got a promotion: project manager/lead engineer. And I was about to lead a team of one: ME.
It’s a nice way of saying “I got volun-told.”
The project involved designing and implementing System Center Configuration Manager (ConfigMgr) and Operations Manager (OpsMgr) for a healthcare company. They wanted to use ConfigMgr for standardizing desktop operating system deployments and OpsMgr for unified data center monitoring.
And while I’d like to think of it as a proof-of-concept activity, the customer wanted it to be a part of their production environment upon completion.
Lesson #1: Set the right expectations.
As the lead engineer, I was responsible for the delivery of the project. But more important, I was responsible for managing risks.
Which means I needed to know what those risks are and communicate them properly to the customer.
Sometimes, customers are open to changing their decisions once they understand the risks.
More often than not, they insist on what they want – regardless of the risks. Especially when contracts have already been signed.
I guess you already have an idea where I’m going with this.
The customer insisted that this implementation be a part of their production environment. And there was no changing their mind.
So, I had to come up with a strategy where part of it will be included in their production environment while some won’t be.
I chose OpsMgr to be the part of their production environment. And I communicated this to the customer. What I didn’t tell them was ConfigMgr will be deployed in an isolated network.
You’ll find out later why.
Lesson #2: HA/DR is a team sport.
I asked the network engineers to create an isolated network to deploy ConfigMgr.
When they asked why, I explained that the isolated network will provide the necessary protection in case “something” goes wrong. They instantly understood the rationale despite not having a slightest clue about ConfigMgr.
I went ahead and built the infrastructure – from the site server, distribution point, software update point, and the SQL Server machine.
Keep in mind, the purpose of deploying ConfigMgr was to standardize deployment of their desktop operating systems. So, I had to build an operating system image that every employee uses.
Once I had the operating system image built and ready for deployment, I asked one of their sysadmins to start monitoring the infrastructure with their existing monitoring tools.
He was a bit confused with my request. “Why monitor the infrastructure when you haven’t built OpsMgr yet? Isn’t OpsMgr supposed to replace our existing monitoring solution?”
His confusion was justified. But I told him what I didn’t tell the project owner: “ConfigMgr is in an isolated network.”
Lesson #3: Mistakes are inevitable. And so are outages.
Testing ConfigMgr meant automating the deployment of a desktop operating system. In this case, it’s Windows XP.
I asked the sysadmin to build demo virtual machines that we can use for testing.
Then, I scheduled the deployment at 6AM. I figured, an hour will be enough to complete the deployment. By then, I will be having coffee and heading over to the office.
The next morning, I got woken up by an email alert at 6:15AM. The message: SQL Server is DOWN.
I was no longer part of the operational support for customer databases at this point. So, I was caught off guard when I received the notification.
The email alert was vague. SQL Server? For what? Which customer? Which environment?
If I was getting the alert, this must be high priority.
So, I logged on to my computer and checked. I was shocked at what I found out.
The Windows XP image that was supposed to be deployed to the demo virtual machines? Well, it got deployed on the SQL Server machine. The database server for ConfigMgr.
I made the mistake of deploying using an incorrect range of IP addresses.
Instead of deploying to the demo virtual machines, it deployed to the SQL Server machine. And it wiped out the entire server.
I was laughing out loud while logging into a Dell PowerEdge server and seeing the Windows XP wallpaper. Hey, at least the Windows XP image deployment was successful.
A few minutes later, I got a call from the sysadmin who configured monitoring. He, too, got the alert. But he couldn’t figure out what the alert meant.
I explained what happened and apologized for my mistake. And instead of panicking about the “outage”, we both ended up laughing.
At the weekly stand-up, I told the project owner the bad news: the SQL Server machine got wiped out. And to make it even more dramatic, I took a long pause before telling him about it.
That “incident” taught me so much about the most important piece in high availability and disaster recovery: PEOPLE
Far too often, we technology professionals fixate on the technology.
And why wouldn’t we? We can geek out on the latest, the greatest, and the implementation details of a specific technology. We can, then, talk about it with other tech folks over beer and pizza at the next user group meeting.
What we don’t realize is that the technology is but a small piece of the entire puzzle.
It’s the PEOPLE who will make decisions…
It’s the PEOPLE who will implement the solutions…
And it’s PEOPLE who will deal with the outage…
It was the project owner who insisted on integrating System Center into their production environment.
It was me who made the call to isolate ConfigMgr from their production network. It was also me who made the mistake of deploying the Windows XP image on to the SQL Server machine.
That “incident” was not really an outage. After all, there was no disruption to normal business operation.
But I felt like it was a great opportunity to highlight the importance we – the PEOPLE – play in planning for and preventing outage situations.
But, the sad truth is…
Just like that “incident”, we humans don’t have the agency to plan for and prevent outages like this.
We are too busy dealing with the usual day-to-day operations. We forget, despite everything going well, that emergencies will happen.
And it’s never a question of “IF”. It’s a question of “WHEN”.
The sad part, we only take action WHEN the unexpected happens. At that point, we don’t have any other choice but to take action.
This incident taught me a very important lesson about high availability and disaster recovery solutions: focus on the PEOPLE.
