Restarting VMs after a Datacenter Down Event

One of my customers recently had a catastrophic thermal event in one of their datacenters and so had to shut down all of their infrastructure at that site.  After the cooling issue was resolved, we were asked to help them to get their infrastructure back online.  Fortunately, we have included several small details as best practices in our vSphere designs, and one of those really paid off for us.  We always create a VM to Host affinity rule that keeps one Domain Controller, the vCenter server, its PSC and its Database (if external) on a known host in the management cluster.

So, after the SAN was powered back on and we restarted the physical ESXi servers, we knew exactly what to do.  I fired up the vSphere client and logged into that ESXi server in the management cluster as root.  From there, I was able to easily find those core infrastructure VMs and powered them all on.  Once they were running, I logged into vCenter... and found that I had an interesting challenge.

We needed to turn back on the rest of the VMs, but we all knew that quite a few VMs in this environment were not running before the event and no one wanted them to be running now.  How to solve this?

Well, it was actually pretty easy.  I wrote up a quick PowerCLI script that checked the vCenter event database for VM Shutdown or Power Off events (some of the VMs did not have VMware tools and so were hard powered off).  Once that list of VMs is collected, the script loops through it and attempts to power on any VM that isn't already running (since, in this situation, some admins had been powering on high impact VMs by hand as soon as vCenter was back online).

It's a pretty simple script, but it worked great for us!  Please bear in mind that this was written in a single morning (under a bit of stress...), so while it worked for us there's no guarantee that it'll work for you.  As always, test thoroughly, use this for educational purposes and, if you fix something, please let me know!

Comments

  1. Darshan KolambkarJuly 6, 2016 at 6:13 PM

    Hi Jason nice script i really appreciate your efforts. I have a doubt if we have a huge infrastructure and try to boot all VMs at a same time we may get hit with boot storm on storage. May be pull information from DB and create batches may help. Please correct me if i am wrong.

    ReplyDelete
    Replies
    1. Jason ColemanJuly 6, 2016 at 8:18 PM

      You are absolutely correct. In this case, the admins had already turned on all of the high priority systems (they got about 50% of their VMs) while I was putting this script together, so it only had to turn on the remainder. In order to avoid a boot storm, just put "Start-sleep 15" or something in the Power On VMs foreach loop; that'll add a 15 second delay between each VM power on operation.

      Delete
  2. Darshan KolambkarJuly 7, 2016 at 1:28 AM

    Perfect 👍

    ReplyDelete
  3. Darshan KolambkarJuly 7, 2016 at 1:29 AM

    Perfect 👍

    ReplyDelete

Post a Comment

Sorry guys, I've been getting a lot of spam recently, so I've had to turn on comment moderation. I'll do my best to moderate them swiftly after they're submitted,

Popular posts from this blog

Clone a Standard vSwitch from one ESXi Host to Another

One of my customers is standardizing the configurations on their HP C7000 enclosures (they've been set up at various sites by various administrators with varying involvement from the architecture team).  As such, we need to stand up some temporary resources so that we can take down the main enclosure for reconfiguration.  That's fine, we can easily ship a smaller enclosure to each site for temporary compute resources.  Some of the sites are using Standard vSwitches, so we need to be able to quickly copy the networking configuration over from their existing blades to these new, slightly different blades.  As I see it, we had 2 good options: 1) Capture a Host Profile with the desired vSwitch configuration.  Delete every other component of the Host Profile so that only the networking section is applied; design the new ESXi hosts so that the vSwitches'll work with the old vmnic-to-vswitch configurations. 2) Write a script to clone the vSwitch from ones ESXi host to another.
Read more

PowerShell Sorting by Multiple Columns

PowerShell's Sort-Object cmdlet is super useful, especially when preparing output for human consumption.  A few people have found my blog while looking for more information about its use, specifically while looking for how to sort by multiple columns (well, properties, technically).  I've never done so (much less written about it), so I hope those folks found answers elsewhere.  But, it got me curious... and it turns out that it's really easy. The Technet article on Sort-Object has the answer directly spelled out: just use commas to create a list of properties to sort by (in order to precedence).  Let's look at some examples!  First, prepare a variable with some good sortable data: $myData = get-eventlog System -newest 25 And then we can get to sorting!  Say you want to sort primarily by EntryType (Warning, Information, Error) and then by Source.  Easy-peasy: $myData | sort EntryType,Source How about if you want it to be in descending order?  Yeah, there&
Read more

Deleting Orphaned (AKA Zombie) VMDK Files

A problem that most Virtualization Administrators need to deal with is Orphaned VMDK files (or Zombie VMDK files, as some people call them).  These are Virtual Machine hard drive files that are just hanging out, taking up storage, but are attached to no VMs.  How can this happen?  Well, the most common way is when an administrator needs to delete a VMDK but isn't quite ready to commit... so, after pressing "remove", they select "remove from virtual machine" rather than "remove and delete files".  The intent is always good: "I want to be able to restore this if it turns out to be necessary... so I'll just come back and delete it next week if nothing breaks!"  But, of course, next week something else happens and the file doesn't get deleted.  After a little while, it turns into "which files did I rename, again?" and nothing seems to happen. So, how do you deal with these orphaned files?  Well, a few years ago I found a scrip
Read more