Virtually Jason

I've had the opportunity to use vRealize Network Insight (vRNI) lately during a network migration project and it has proven invaluable.  We've used it to collect data about the subnets before they're migrated and we use it to help troubleshoot issues after the migration is completed.  It's given us great visibility into the traffic on the network and into where that traffic is being blocked.  So, how do we use it? Before the migration, we use it to scrape a ton of data from the source subnet, as we need to know what's going on with the servers that are running there.  At the start of the project, we attempted to learn those details by asking the application owners about their applications' requirements, however we found that the vendor documentation was universally poor, especially when compared against the needs of micro-segmentation. To get that information, I execute a very simple query in vRNI: flows where subnet = <subnet> .  This returns a list o...

We’ve all probably been in this situation: vCenter has experienced a catastrophic failure and isn’t even responding to ping, so you need to get console access in order to resolve the issue.  The challenge is that the VM could be running on any of 16 ESXi hosts in your cluster.  Generally, I like to create DRS Host Affinity rules for the vCenter and its SQL server so that I know which host those VMs should be running on during normal operations, but if HA has been busy they could still be anywhere.  That means that you need to find those VMs in order to troubleshoot, which could take a while.  Instead of opening 16 vSphere Client instances, each connected directly to an ESXi host in your environment, there’s a much easier way to find those VMs: PowerCLI! PowerCLI can query a whole array of ESXi servers fairly quickly, returning the host(s) that you need to log the vSphere Client into in order to fix things.  To do this, you’ll need to change your PowerCLI insta...

How often do we, as system admins, find ourselves with a strange error that might be due to network connectivity?  Pretty often, in my experience.  While Ping is a great tool for validating basic routing, we live in a world of firewalls – be they physically on the network, virtually on the network, or software installed on the server.  When diagnosing network connectivity issues, Ping is not good enough, as that only tests the ICMP network flow.  You can’t ping a specific port in Windows, so if you’re looking to test something like HTTPS (TCP 443), with Ping you’re out of luck.  Fortunately, there are many other tools available. My favorite tool is PortQry  (that link will work as long as Microsoft graces us with a consistent URL… so until tomorrow, probably).  PortQry was originally a SysInternals tool (developed for Windows 2000!) and is still one of the best.  You’ll have to download it, but it’s a simple executable that you can invoke fro...

A few customers have been hit by an intermittent issue where virtual machines seem to reject their network adapters.  In Windows, this shows up as the guest OS reporting a hardware failure on the NIC (which, given that the NIC is virtual, is bit of a hard sell).  So, while the VM has a network adapter attached to it and it is connected to the network, the VM doesn’t get any network access.  If you open up device manager, the “general tab” for the network adapter will show an error (error 10 if memory serves).  When you try to update the driver on the NIC, it will fail to install it.  I’ve only ever seen this with the vmxnet3 adapter, but I’ve heard that it can affect e1000 as well. Typically, the VM’s network connectivity can be restored by some combination of removing the vNIC from the VM and adding it back (well, it’s technically a new one as it will have a new MAC address), reinstalling the VMTools and/or removing nonpresent NICs from the guest OS.  T...

A customer recently asked me how to easily check their environment for high CPU Ready metrics.  Given that I don't know of an easy way to quickly look through the whole environment by the GUI, I decided to pull out PowerCLI and see what I could put together. For those who don't know, CPU Ready records the time that a VM spends (during a given interval) ready for CPU time, but no physical CPU is available.  Basically, it reports on CPU contention.  There will always be small amounts of CPU Ready - that just reflects the time that it takes a CPU to react to an OS's request to compute.  In general, CPU Ready times averaging below 5% of your polling interval should be completely unnoticeable.  Between 5% and 10%, delays become noticeable, and beyond 10% they are seriously problematic. Those rules for interpreting CPU ready are all based on percentage of a time interval, but the metric that is actually recorded is the number of milliseconds spent ...