Virtually Jason

One of my customers was doing some work on their storage network and so wanted a list of all ESXi hosts and their WWNs.  This struck me as another excellent scripting opportunity, so I pulled out my old Brocade 1:1 Zoning script and went to work.  I really only needed a one-liner for this situation, so I extracted my needed syntax from that script and quickly put it together.  Here's the (admittedly, ugly) command that I came up with: get-vmhost | select name,@{N="WWNs";E={($_.ExtensionData.config.storagedevice.hostbusadapter.PortWorldWideName | % {("{0:X}"-f$_ -split '(..)' | ? {$_}) -join ':'}) -join ", "}} That'll spit out a list with two columns, ESXi host names in one, and a comma separated list of WWNs in the other.  So, what's that ugly expression in that select statement doing?  Well, it's collecting all of the port WWNs on that ESXi host, then converting each one from decimal into hexadecimal (the "{0:X}...

One of my customers had a whole collection of ESXi 6 hosts that were all installed from the April 2016 HP ESXi ISO image... and they hadn't been patched since then.  Well, one day they called me because their Splunk server had started sending out alerts from an alarm that we'd set up to monitor the ESXi hosts for memory leaks. So, I logged into one of the affected hosts to try and figure out what was going on.  After poking around in a bunch of logs and a fair amount of google work, I came across this article about a memory leak over at CPU Ready.  It sure looked promising.  So, I followed their instructions to check the version of the broadcom driver in one of the affected ESXi hosts and, sure enough, it was an older version.  Fortunately, HP has a fix available, so I just needed to get it installed on all of the ESXi hosts (since full on ESXi patching wasn't necessarily available, unfortunately). I needed some way to figure out exactly which hosts needed ...

One of my customers recently experienced a strange issue.  One of their ESXi hosts had entered a problem state where Storage vMotion and vMotion were failing for all VMs on the host (vMotion was failing at 13%, which is an interesting spot).  We initially noticed the issue when Storage vMotion repeatedly threw an error for one of their VMs: A general system error occurred: Failed to create journal file provider: Failed to open "/var/log/vmware/journal/..." for write: There is no space left on the device. Well, that error seemed self explanatory.  I connected to the host's CLI (SSH was already enabled, which made that easier) and did a "vdf -h" to look at its file system.  I was surprised to find that none of the partitions were full, so I dug deeper. I decided to take a look at the vmkwarning log file, which is frequently a gold mine when troubleshooting ESXi host issues.  So, I did a quick "tail /var/log/vmkwarning.log" and, lo and  behold, we h...

One of my customers set up a large number of ESXi hosts, installed on local storage.  Since they're not doing anything with that storage, they never bothered renaming those datastores and so their vCenter inventory was filled with instances of "datastore1 (#)".  This is just a housekeeping thing, but that's really messy.  Generally speaking, a bit of mess isn't a big deal, but those little messes always seem to cause problems when you're troubleshooting some other major issue.  And, fortunately, this is really easy to fix.  Rather than write my own script, I did some quick googling and found that VMNICK had done all the work already, with his own " rename local host storage " script. In his post, he notes that the script might explode if you use the string "datastore1" on any other datastores in the environment.  Excepting a very particular subset of situations, I'm not a fan of explosions, and so I figured that I could probably fix t...

One of my customers wanted to reset the root password on all of their ESXi servers.  This can be easily done through a Host Profile (which is particularly useful if you don’t know the root password), but that would require putting the hosts into maintenance mode and waiting for all of those vMotions to apply.  There’s no need to put a host into maintenance mode just to change the root password, and so we looked for a better solution. Jeff  found the PowerCLI cmdlet, set-vmhostaccount , and with that, we went to work.  The script is actually really simple – it just connects to the ESXi host, then uses the set-vmhostaccount cmdlet to change the password, and records the change. For flexibility, I included a few variables as parameters.  You may want to select only a subset of ESXi servers, and so there are both -hostSuffix and -hostPrefix parameters.  If the hosts that you want to target are all called Sac-ESX#.lab.local and the production hosts that you’...

Memory overcommitment remains a contentious topic, despite some really great studies on the topic.  Regarding VDI, I’ve heard opinions ranging from “0 memory overcommitment!” to “200%, 300%, it’s fine!”  I figured that I’d share my thoughts on the topic and see if anyone else wants to weigh in on the discussion.  First though, some points about the decision. The main argument that I’ve heard against memory overcommitment boils down to protecting the user experience.  Since virtual desktops have users actively logged into and using them, even slight performance degradation is immediately noticed.  We all know that hypervisor swapping is Very Bad for performance.  If you don’t overcommit your hosts’ memory, you won’t have swapping, so why overcommit? The simple answer is that, while 2 GB of RAM for a single desktop is no big deal, 2 GB of dedicated RAM for 1000 desktops is a lot harder to accept.  Memory overcommitment can be an important technique for...

A few customers have been hit by an intermittent issue where virtual machines seem to reject their network adapters.  In Windows, this shows up as the guest OS reporting a hardware failure on the NIC (which, given that the NIC is virtual, is bit of a hard sell).  So, while the VM has a network adapter attached to it and it is connected to the network, the VM doesn’t get any network access.  If you open up device manager, the “general tab” for the network adapter will show an error (error 10 if memory serves).  When you try to update the driver on the NIC, it will fail to install it.  I’ve only ever seen this with the vmxnet3 adapter, but I’ve heard that it can affect e1000 as well. Typically, the VM’s network connectivity can be restored by some combination of removing the vNIC from the VM and adding it back (well, it’s technically a new one as it will have a new MAC address), reinstalling the VMTools and/or removing nonpresent NICs from the guest OS.  T...

Have you ever had a full datastore and needed to easily discover which folder was the main culprit?  I’m sure that there are a thousand ways to do this, but one easy one is available from the ESXi host command line.  Either pull up the local console (with the local Shell enabled) or open an SSH session remotely (I know, we’re not supposed to do that anymore!).  Then, navigate to the datastore that you wish to examine with a command like: cd /vmfs/volumes/MyDatastore Once you’re on the datastore, use the “du” command to display disk usage.  If you use the command by itself, it will recursively display the disk usage of all folders/subfolders from your current location in kilobytes.  While that’s useful, it’s not the most user friendly thing in the world.  Instead, I like to use this command: du –m –d 1 That will display the disk usage in megabytes (it would be nice if it had a gigabytes option, but oh well) and will only display the total s...

I’ve been working with Unidesk lately, as well as with the vSphere PowerCLI .  One of my customers has also been having an intermittent issue where they cannot browse their datastores (it just sticks at “Searching Datastore…” until they lose patience and close the window).  All of this has gotten me thinking about the relationship between vCenter and the ESXi hosts.  When we send a command to vCenter, such as creating a new VM, we all know that vCenter is simply relaying that command to one of its ESXi hosts, which does the actual work.  Same thing with file copies through the Datastore Browser, and in fact even reading the datastores through that same tool. How do we, as administrators, know which ESXi Host vCenter is using as its “workhorse”?  In a clustered environment, it’s really very unclear.  It turns out that, if that workhorse ESXi host is having an issue, it may prevent a lot of those basic vCenter functions from working (such as the aforem...