Memory Leak on the April HP ESXi Image

One of my customers had a whole collection of ESXi 6 hosts that were all installed from the April 2016 HP ESXi ISO image... and they hadn't been patched since then.  Well, one day they called me because their Splunk server had started sending out alerts from an alarm that we'd set up to monitor the ESXi hosts for memory leaks.

So, I logged into one of the affected hosts to try and figure out what was going on.  After poking around in a bunch of logs and a fair amount of google work, I came across this article about a memory leak over at CPU Ready.  It sure looked promising.  So, I followed their instructions to check the version of the broadcom driver in one of the affected ESXi hosts and, sure enough, it was an older version.  Fortunately, HP has a fix available, so I just needed to get it installed on all of the ESXi hosts (since full on ESXi patching wasn't necessarily available, unfortunately).

I needed some way to figure out exactly which hosts needed this new driver version installed, and since this was only an updated driver rather than an updated ESXi version, it wasn't necessarily easy to tell which hosts had the problem driver and which didn't.  This problem was resolved in the October 2016 HP ESXi ISO image, which installs the same build of ESXi, just with this newer driver.

The instructions in the article that I followed described using ESXCLI to discover the driver version, but I sure didn't want to enable SSH on every ESXi host, connect to each host, and check its driver version.  Fortunately, PowerCLI has an ESXCLI cmdlet, so I didn't have to.  I just used this command to output a list of all ESXi servers in the environment and which version of the driver each was using:

get-vmhost | foreach {echo "$($_.name): $(((get-esxcli -vmhost $_).network.nic.get('vmnic0')).driverinfo.version)"}

In our case, all of the hosts that were using version 2.712.70.v60.3 of the driver were using the version from the April image and so needed to be updated, whereas the hosts using version 2.713.10.v60.4 were from the October image and so were fine.

Edit:
In case you want this output in a more PowerShell friendly format, you can use these commands instead:

get-vmhost | foreach {
    $outObj = "" | select vmhost,driverVersion
    $outObj.vmhost = $_
    $outObj.driverVersion = ((get-esxcli -vmhost $_).network.nic.get('vmnic0')).driverinfo.version
    $outObj
}

Comments

Post a Comment

Sorry guys, I've been getting a lot of spam recently, so I've had to turn on comment moderation. I'll do my best to moderate them swiftly after they're submitted,

Popular posts from this blog

Orphaned VMDK Files

Migrating from one vCenter to Another, Improved

Deleting Orphaned (AKA Zombie) VMDK Files