Validating HA by Inducing a PSOD

After installing an ESXi cluster, of course you want to validate its functionality.  vMotion/DRS are easy to validate, but how about HA?  Just pull the power cable out of the back of one of your ESXi hosts, right?

Well, that will do it... but it's also a bit risky.  I've seen host configurations fail to come back from an unexpected power failure and I've heard stories of hardware damage stemming from such an event.  So, a power failure is probably not the best option... so what else can you try?

You could always induce a Host Isolation event.  Simply disconnect all of the network cables from the ESXi host, which will trigger the specified HA response.  That works, but it's a bit of a pain, as you're likely going to need to unplug several cables and keep track of exactly where they belong.  If you're remote, you'll need to involve the network team to get them to shut down specific ports (and you'd better hope that there's no miscommunication regarding which ports you're working with).

I'm not a big fan of that approach either.  Fortunately, there is an easy option that I've had good experience with: inducing a Purple Screen of Death.  From the command line of an ESXi host (either the local console or an SSH session), there is a command to cause a PSOD.  This is awesome, as it allows you to both verify that HA is working and validate that your scratch/core dumps are being written to persistent storage.  The process is pretty simple,  but before you do this, make sure that you've got some way to restart the server, as it is going to drop off the network ;)

I like to run two commands, after disabling DRS.  First, I use:

esxcli vm process list

This lists the running processes of every VM on the ESXi host.  This is where I make sure that I don't have any production workloads running on the ESXi host and that I have the expected test VMs on there.  Once I'm confident that this is the correct ESXi server and that I'm not going to impact production VMs, I use this command to cause the actual PSOD:

vsish -e set /reliability/crashMe/Panic 1

And that's that.  After issueing that command, you'll either see your PSOD or your SSH session will terminate.  In about 30 seconds, your VMs will start failing over to remaining hosts within your cluster and they should be back online within a few minutes.  Once you're sure that everything's good, reset the "failed" ESXi host and then it's back to business as usual.

Comments

Post a Comment

Sorry guys, I've been getting a lot of spam recently, so I've had to turn on comment moderation. I'll do my best to moderate them swiftly after they're submitted,

Popular posts from this blog

Clone a Standard vSwitch from one ESXi Host to Another

One of my customers is standardizing the configurations on their HP C7000 enclosures (they've been set up at various sites by various administrators with varying involvement from the architecture team).  As such, we need to stand up some temporary resources so that we can take down the main enclosure for reconfiguration.  That's fine, we can easily ship a smaller enclosure to each site for temporary compute resources.  Some of the sites are using Standard vSwitches, so we need to be able to quickly copy the networking configuration over from their existing blades to these new, slightly different blades.  As I see it, we had 2 good options: 1) Capture a Host Profile with the desired vSwitch configuration.  Delete every other component of the Host Profile so that only the networking section is applied; design the new ESXi hosts so that the vSwitches'll work with the old vmnic-to-vswitch configurations. 2) Write a script to clone the vSwitch from ones ESXi host...
Read more

PowerShell Sorting by Multiple Columns

PowerShell's Sort-Object cmdlet is super useful, especially when preparing output for human consumption.  A few people have found my blog while looking for more information about its use, specifically while looking for how to sort by multiple columns (well, properties, technically).  I've never done so (much less written about it), so I hope those folks found answers elsewhere.  But, it got me curious... and it turns out that it's really easy. The Technet article on Sort-Object has the answer directly spelled out: just use commas to create a list of properties to sort by (in order to precedence).  Let's look at some examples!  First, prepare a variable with some good sortable data: $myData = get-eventlog System -newest 25 And then we can get to sorting!  Say you want to sort primarily by EntryType (Warning, Information, Error) and then by Source.  Easy-peasy: $myData | sort EntryType,Source How about if you want it to be in descending or...
Read more

Deleting Orphaned (AKA Zombie) VMDK Files

A problem that most Virtualization Administrators need to deal with is Orphaned VMDK files (or Zombie VMDK files, as some people call them).  These are Virtual Machine hard drive files that are just hanging out, taking up storage, but are attached to no VMs.  How can this happen?  Well, the most common way is when an administrator needs to delete a VMDK but isn't quite ready to commit... so, after pressing "remove", they select "remove from virtual machine" rather than "remove and delete files".  The intent is always good: "I want to be able to restore this if it turns out to be necessary... so I'll just come back and delete it next week if nothing breaks!"  But, of course, next week something else happens and the file doesn't get deleted.  After a little while, it turns into "which files did I rename, again?" and nothing seems to happen. So, how do you deal with these orphaned files?  Well, a few years ago I found a scrip...
Read more