So… What’s YOUR disaster recovery plan?
So you bought a nice server or two. You even sprung for a tape auto loader. Nice, right? Your backup software sends you an alert each night. You even take a tape offsite once in a while. You’re good, right? You’re totally ready for any disaster, right? WRONG! Tape is nice, but tape is only one part of a comprehensive plan. And the way things are going… it’s becoming a smaller and smaller part each day. Wha, wha, what?!? Read on…
Wait, what the heck is wrong with tape again? Well, I’ll tell you. First lets affirm again that tape is good. It’s a mature technology, it’s fast, portable and robust. And secondly, let’s define what type of environment we’re talking about here. Many of my customers are small businesses. Many have only one server. Almost none have a full-time IT person, and nobody (other than me) EVER told them that they should TEST their backup systems. And nobody (other than me, maybe) EVER told them to do it routinely. And so, they are, in a sense, flying blind without a net when it comes to Disaster Recovery.
Wait a second, is this some kind of trick? You still haven’t said what’s wrong with my tape system. No, no I haven’t, but I will. One of the first things you figure out when you’re thinking about disaster recovery is… you have to define the scope of the disaster you’re preparing to defend and or recover from. So let’s make it simple. Let’s just assume that your server blows up. Ok, maybe it just melts down. The server, your server, is not A server anymore. And to boot, the water that was used to put out your server fire… it shorted out your auto-loader (and all of the tapes in the unit). Let’s just assume that’s all that happened. Whew! Dodged a bullet. At least your switch and firewall and phone system and HVAC and access control systems are all spared! Now what? Good thing you took that tape offsite last weekend. You’ll only loose 4 days of data. Wait, that’s good? Well, I suppose it is if you consider that you would have lost everything without that backup. Let’s hope it was a good backup.
Now that we have the disaster scoped… what are you going to do from here? You feel pretty good… you’ve got a tape. And then you start wondering if all of your data was actually backed up? Hmmm, probably is OK. Moving on. What do you do? You have no server, and you have no auto-loader. You get on the horn with your IT guy who get’s on the horn with Dell. He can get a replacement server built and shipped in two weeks. Then he’ll have to configure it… and then… uhh, better get that auto-loader ordered too. Good news, that’ll get here at about the same time. Wait, your office is going to sit around with no email, accounting system, work instructions… nada, nothing for TWO WEEKS. But at least you have your data on that tape. Hey, at least 4 day old data won’t matter too much after a two week wait. That’s when your IT guy makes some calls and finds a local vendor with a system that should work. It’s twice as much, but you can get it in two days! It’s crisis time so you go for it. And to boot, you keep the Dell order coming, as you’re not going to be in this situation again. You’ll have two servers!
So two days later… the IT guy shows up with the server and auto-loader, you hand him the tape and he asks, “where’s your <backup vendor> Disaster Recovery Disc?” You want what now? You know, the disc that the instructions say you should create? It’s bootable… let’s you recover. Oh, and he wants that special file that the instructions tell you NOT to keep on your system, as you’ll need that to recover too. My what now? Yeah, there were some things in there that someone should have been keeping track of… Bummer. No problem, we’ll call the vendor for some help. Bad news, one of those things you should have been keeping track of was your licensing… and your maintenance expired last October. Good news though, they’re happy to help at their emergency support help rate. Plus you have to purchase the complete software package again at the full MSRP. Ouch. Of course this is fiction, but what really happened is… the backup software vendor sold that product to another vendor. THAT vendor rebranded and updated the product while you weren’t paying attention. And so the simple call to the vendor for help took most of a day trying to find the right folks and explain your situation. But who’s keeping track of the details?
Flash forward to three more days later… you and your IT guy finally get your new and expensive server up and running. You fought for 36 hours straight trying to restore your backup to a completely different server make and model. And to complicate matters you didn’t think through the drives and drive space when you ordered it. So your installed programs had to be remapped to different drive letters… just about everything that could have gone wrong, did. None of the drivers installed on the original server matched the new server. Every step was painful. It was, well and truly, a nightmare. Oh, and I forgot to mention that the IT guy that was helping you… well, he wasn’t the original IT guy that setup your original system. That little tidbit was actually more painful than all of the missing, disparate and mismatched hardware that you had to cobble together on which to restore. Amazingly, you are back up and running in one business week, with data that is now a week and half old. And you’re HAPPY to be there.
Are you depressed yet? I am. These are the kinds of things that IT guys (and gals) deal with in the small business market. You THOUGHT you were good. You THOUGHT you set everything up correctly. You THOUGHT you were getting good backups. Daggg! You had your tape! And this is one scenario that exposes what is wrong with tape. Tape requires a very specific environment to be useful. It also exposes just how much complexity and inter-connectivity and cooperation is actually required for a comprehensive disaster recovery plan.
It’s a good thing that you’ve learned from this fictitious scenario, right?. =) Sure you did. So do you want to try and pre-solve all of the potential mis-steps outlined above? Admirable, but… there’s a better way. Guess what? It doesn’t involve tape! Wait, but your tape saved you above, right? Eventually. Sort of. I guess. But there’s a better way. And it won’t break the bank. Do tell.
Virtualize. What? I thought we were talking about tapes? Listen up… the tape isn’t the problem in the above scenario. The problem is that your functional server is tied to a specific piece of hardware: your <vendor> server. And to boot… your server is only backed up by yet another specific arrangement: your <vendor> tape drive. A better mouse trap wouldn’t be a “trap” at all. A better solution is to make sure that your system backup and data can be restored easily and universally. Hmmm… some ubiquitous combination of technology? Something that would allow you to restore to almost ANY hardware. And hardware that isn’t specialized… like your tape drive. Sounds expensive.
Virtualize. There it is again… you keep saying that word. That is the key. Here’s what you need to get your head around in order to see the magic. See the power. Can you feel it? Instead of thinking of your server as that physical piece of equipment in your phone room, closet or bathroom (yes, I have a client with a server, actually three, in his employee loo). Think of the “server” as the function that it provides. Mostly that’s going to be file storage, and maybe it’s going to be some applications… QuickBooks, Exchange… maybe. If you’ve purchased a “server” in the last 5 years, odds on favorite that you have a system that is way more powerful than you actually need. And using virtualization you can more efficiently use that extra power or capacity. With enough RAM, a standard server can easily host several virtual servers. But in our scenario we only have one server, right? We don’t need to virtualize, right? WRONG! Were you paying attention up there? Did you already forget about all of the problems we encountered? Virtualization is how we can avoid every single one of them!
When you virtualize a server, you basically define a box that server lives in. Not a physical box but a virtual box. Sounds magical. In the Windows world the box is defined by Hyper-V. Hyper-V is included in the core operating system of 2008, 2012 and now 2016 Server. Personally, I prefer 2012 right now (it’s OK to NOT be an early adopter if you don’t have a compelling reason to risk it). In our scenario we would fully install and patch Windows 2012 R2 Standard on our new or old Dell server. This host installation is the only “server” that even knows it’s on a Dell server. Our “functional” server then gets installed into a virtual box (that we define). We tell it how how much RAM it gets, how much hard drive space, how many processors, network cards, etc. Once that’s done we simply install Windows 2012 R2 again, but this time into that virtual box. From this point on that new virtual server can ONLY be accessed via a Remote Desktop window.
Why do you keep going on about this virtual box business? Because, once you “get” that part then you’ll realize that ANY server (be it Dell, HP, clone…) running 2012 R2 Standard with Hyper-V installed can now define the same virtual box. And, indeed, can then run our “functional server! We have successfully insulated or abstracted our “server” from the hardware layer. We are no longer dependent on any specific brand or type of equipment. As long as it can run the host operating system and has the resources required (RAM, hard drive space, etc.), then we’re golden. We’re up and running. Boom done.
So we’re done then? Uhhh… no. I’m just getting warmed up. But the important step to grasp is the virtualization (the magic box). If you’ve “got” that step, then now we can tie it all together. Yeah! What about my tape? No tapes! With some clever backup software that is aware of the virtual machines we can simply backup across the network to a NAS (network attached storage) AND to normal… wait for it… COTS (consumer off the shelf) USB hard drives. You know, like the ones you see at Costco? $100 for 3 TB drive with no external power supply. Those kind. That doesn’t sound safe, is it? Why yes, yes it is. The primary backup is still onsite to the NAS. With a very moderate NAS device we can usually hold 6 months of daily backups. If that NAS is housed in an alternate location in the facility (safe, of course) then it can protect your data against local catastrophes (as described above). Each night after the primary backup to the NAS the backup software then commits the same backup to an “offsite” USB drive. It’s not offsite yet, but it’s intended to be taken offsite each day (preferably). Wait, that doesn’t sound safe. Well, the drive is encrypted. Basically without the encryption key (think… long password) the drive is just a hunk of plastic, rubber and metal. Well, actually it could be reformatted and used by others, but they would never, ever get to your data. Evah.
Well that sounds pretty good. Maybe you’re sold? Not so fast! It gets better and better. Now that you have TWO servers (remember the one that you ordered that was going to take two weeks?) you can configure the second server just exactly like the first server (Windows Server 2012 R2 with Hyper-V installed). Why do I need a second virtual server? The second server can be setup as a replication server. Once the two servers “know” about each other, they can replicate or synchronize with each other. Your functional server living in the virtual box can be copied to the second server. What like once a day or something? No, like every 15 minutes! If your primary server chunders (for any reason)… maybe you just want to take it offline to do maintenance… then you can bring up the functional server on the second server. Nobody’s the wiser either. You can even do it without bringing the functional server offline… if you want. So… that’s a lot of hardware for one virtual server. True, but I didn’t tell you to buy it. =) Actually, one nice thing is that you can segregate your server roles that would normally be lumped all onto that one physical server into more logical virtual or functional servers. And so, it’s nice to have an accounting server with QuickBooks on it that is separate and distinct from the main file share server. That way QB updates can be applied, and the server rebooted without affecting the other server functionality… at all. Now, once you get there… you can actually host the accounting functional server on the second server (with the original server minus QB running on the original server). And yes, you can still replicate from both servers to the other. That way both servers hold a functional or near functional replica of both servers. Bonus! less load, twice the safety! Can’t get much better than that! Well.. it could, but let’s stop there!
So, there you have it… sure it wasn’t concise, but it had teeth, didn’t it? There are some takeaways. Virtualization… good. Tape… not quite as good as it used to be. Not paying attention… really bad. Does any of this ring true in your organization? Need someone to pay attention for you? Give me a ring; it’s what I do. Thanks for listening.
PS- If you find this style of writing more engaging than a more dry and less chatty tome… please let me know (either way).