Virtually Jason

Several of my customers use HP C7000 Blade Chassis for their ESXi hosts.  I've picked up a few tips and tricks for working with that chassis over the years, so I figured that I'd post them here. The Virtual Connect (the blade chassis's networking component) has a feature that can prevent pause frames from flooding a network by disconnecting a blade that is sending an excessive number of them.  Unfortunately, every now and then, it detects an ESXi host's uplink as sending such a number of pause frames and so disconnects that network adapter.  Fortunately, it's really easy to allow traffic to flow through that port once again.  Just SSH into the Virtual Connect (you can get the address by looking at the "Virtual Connect Manager" link in the Onboard Administrator interface.  Once you're connected, use the show port-protect  command to see if there are any ports that are in a blocking state.  If so, you can use the reset port-protect  command to res...

One of my customers recently came across an issue where their ESXi hosts were randomly crashing with a PSOD.  They had recently applied the latest SPP from HP and the latest ESXi 6.0 patches, and were now occasionally seeing these crashes with messages like "LINT1/NMI (motherboard nonmaskable interrupt), undiagnosed.  This may be a hardware problem..." As the PSOD implied, they had called HP support for help, but weren't making much progress.  I did some googling and found a really interesting blog post from Jason Whitelock about a recent ESXi update causing HP servers to PSOD .  He had come across the exact same issue and had tracked it down to the value of the iovDisableIR setting, which had changed in this latest ESXi update.  When he set it back to its original setting, the PSOD issue went away. As VMware explains it, Interrupt Remapping (the technology that's affected by this setting) enables more efficient IRQ routing and thus improves performance...

One of my customers had a whole collection of ESXi 6 hosts that were all installed from the April 2016 HP ESXi ISO image... and they hadn't been patched since then.  Well, one day they called me because their Splunk server had started sending out alerts from an alarm that we'd set up to monitor the ESXi hosts for memory leaks. So, I logged into one of the affected hosts to try and figure out what was going on.  After poking around in a bunch of logs and a fair amount of google work, I came across this article about a memory leak over at CPU Ready.  It sure looked promising.  So, I followed their instructions to check the version of the broadcom driver in one of the affected ESXi hosts and, sure enough, it was an older version.  Fortunately, HP has a fix available, so I just needed to get it installed on all of the ESXi hosts (since full on ESXi patching wasn't necessarily available, unfortunately). I needed some way to figure out exactly which hosts needed ...

One of my customers called me for some help troubleshooting a backup issue.  For some inexplicable reason, their VM based backup solution was failing for a bunch of VMs on a specific ESXi host.  When I got there, the first thing that I checked was the host tasks and events.  It listed a whole bunch of failed vMotion attempts for one particular VM with no VM Tools installed, so I thought that I'd take a peak at the VM console to see what I could see.  That failed, with a fairly generic message: "Unable to connect to the MKS: Connection terminated by server" 9 times out of 10, that error indicates that there's a firewall between the ESXi host and the client.  It turns out that this was 1 time out of 10, because some subsequent network troubleshooting revealed that there was nothing odd going on in that space.  My next troubleshooting step was pretty obvious; check the ESXi host logs to see if anything stood out.  So, I logged into the local console of...

Several of my customers use HP C7000 blade chassis for their ESXi hosts.  One of them asked me to help them update the firmware in their enclosures.  There are basically three different pieces of the blade chassis; each one needs updating.  The blades themselves need updating, the Interconnect Modules (basically, embedded switches) need updating and the Onboard Administrator Modules need updating.  There are different and varied steps for updating each of those components.  The blade is updated by bootstrapping with an SPP (Service Pack for Proliant) ISO.  The OA is updated through its own web interface by uploading a specific .bin file.  The Virtual Connect (which is the centralized management for all of the Interconnect Modules) is updated through its own special command line utility by uploading its own specialized .bin file. All of that is a little awkward, but it’s not really that big of a deal.  We did run into one spot of difficulty, as ...