Useful VRNI Queries

I've been working with a large customer for a while, and one of the things that we've done here is to thoroughly integrate vRealize Network Insight into their environment.  I'm not sure that I can go back to using just plain vCenter after this, because vRNI gives me such a cool perspective on the environment!  As I've worked on various situations, I've found myself using a few queries over and over again, so I figured that I'd share some of my favorites!

Queries for Troubleshooting Network Communications Between VMs

  1. Flows where VM = <VM Name>
  2. Flows where Firewall Action = Deny
  3. Flows where Security Tag = <Security Tag Name>
  4. VM <VM Name> to VM <Other VM Name>
These queries (and permutations thereof) are incredibly useful when troubleshooting communication issues between VMs.  Depending on how precisely the application owner can describe their issue, I might use any one (or combination) of these.

For example, if an application owner tells me that ApplicationServerX is unable to connect to DatabaseServerY, I might start by looking at the path between those two VMs.  So, I'd use VM ApplicationServerX to VM DatabaseServerY, and make sure that there is a path between those two systems.  The diagram that this generates also shows all of the firewalls that exist between those two entities, and each of those firewall icons can be clicked upon to take a peak at the applicable rules.  This might be enough for me to figure out why the two systems aren't communicating, but if the firewall rules are complex or if the application owner doesn't really know what ports are needed, I have other tools available to me!

To drill deeper, I might use a combination of #1 and #2 above: Flows where VM = ApplicationServerX and VM = DatabaseServerY and Firewall Action = Deny.  There's a lot to unpack there, but that query is basically going to pull up every network flow that goes between those two servers (regardless of direction) that has been denied by my firewall (or any firewall that's plugged in to vRNI).  This can be a really easy way to see that (for example) TCP 1521 is being denied between those two systems and, if you click on that denied flow, you can even see what firewall rule is causing that deny!

But, what if the application owner is less technical or just can't tell which servers are specifically having the issue?  Well, #2 on that list is a great option!  Because I've applied NSX Security Tags to all of my VMs based on the application that they serve, I can run Flows where Security Tag = Application and Firewall Action = Deny to see all denied traffic flows in that whole application!

It's worth noting here that there is a weakness to troubleshooting in the way with vRNI: its logs are not real-time.  If this is a historic issue (it's been going on for 20 minutes or more), vRNI will have no problem finding it.  But, you're not going to see real-time data in vRNI, and you're always going to see historic data (based on your selected timeframe).  What that means is that, after using the techniques above to find and resolve an issue, when you run that query again, you're still going to see those flows as being denied.  They'll continue showing up that way until the denies have "aged out", based on the timeframe that you are displaying within vRNI.  So, if I'm looking at 1 hour of data in vRNI, those denies are still going to show up in there for a solid hour after I've solved the problem.

Browsing the Inventory

  1. VM where IP Address = <IP Address or subnet in CIDR>
  2. VM where Incoming Port = <Port number>
  3. IPSet where IP Address = <IP Address>
  4. VM where Disconnected Vnic Count > 0
This is another area where VRNI has really shined for me.  vCenter has all sorts of useful data, but it's not always easy to get to.  You can search for a VM based on IP Address, but it's buried really deeply in the menu system.  In vRNI, it's just another query!  What if I want to see all of the Oracle servers in my environment?  That's where VM where Incoming Port = 1521 comes into play!  

In NSX, we frequently define IP Sets to allow us to make policy rules that involve various physical devices.  As the environment grows, it can become difficult to remember which IP Set contains which address (or if one even exists for that address yet!).  IPSet where IP Address = <my IP> is a great tool for that situation, especially when troubleshooting if you're not entirely sure what someone might have named that IP Set!

I included that last example, VM where Disconnected Vnic Count > 0, as an example of the cool queries that you can create in vRNI.  Several years ago, I remember that I put together a PowerCLI script for a customer to look for exactly this information.  It wasn't an overly complicated script, but it still involved several cmdlets and some logic.  Now, I can just use a simple query in vRNI to get those same results!  When typing out queries in vRNI, it auto-populates for you, so if you type VM where and hit space, it'll show you some options... like, a lot of options.  That's what I did, and browsing through the list, Disconnected Vnic Count immediately stood out to me as a cool filter for my query.  But, there are a ton of different things you can use, giving you all sorts of neat ways to explore your inventory.  Also, remember that these can be combined through the operators and or or to put together some really cool queries!

Comments

Post a Comment

Sorry guys, I've been getting a lot of spam recently, so I've had to turn on comment moderation. I'll do my best to moderate them swiftly after they're submitted,

Popular posts from this blog

Clone a Standard vSwitch from one ESXi Host to Another

One of my customers is standardizing the configurations on their HP C7000 enclosures (they've been set up at various sites by various administrators with varying involvement from the architecture team).  As such, we need to stand up some temporary resources so that we can take down the main enclosure for reconfiguration.  That's fine, we can easily ship a smaller enclosure to each site for temporary compute resources.  Some of the sites are using Standard vSwitches, so we need to be able to quickly copy the networking configuration over from their existing blades to these new, slightly different blades.  As I see it, we had 2 good options: 1) Capture a Host Profile with the desired vSwitch configuration.  Delete every other component of the Host Profile so that only the networking section is applied; design the new ESXi hosts so that the vSwitches'll work with the old vmnic-to-vswitch configurations. 2) Write a script to clone the vSwitch from ones ESXi host...
Read more

PowerShell Sorting by Multiple Columns

PowerShell's Sort-Object cmdlet is super useful, especially when preparing output for human consumption.  A few people have found my blog while looking for more information about its use, specifically while looking for how to sort by multiple columns (well, properties, technically).  I've never done so (much less written about it), so I hope those folks found answers elsewhere.  But, it got me curious... and it turns out that it's really easy. The Technet article on Sort-Object has the answer directly spelled out: just use commas to create a list of properties to sort by (in order to precedence).  Let's look at some examples!  First, prepare a variable with some good sortable data: $myData = get-eventlog System -newest 25 And then we can get to sorting!  Say you want to sort primarily by EntryType (Warning, Information, Error) and then by Source.  Easy-peasy: $myData | sort EntryType,Source How about if you want it to be in descending or...
Read more

Deleting Orphaned (AKA Zombie) VMDK Files

A problem that most Virtualization Administrators need to deal with is Orphaned VMDK files (or Zombie VMDK files, as some people call them).  These are Virtual Machine hard drive files that are just hanging out, taking up storage, but are attached to no VMs.  How can this happen?  Well, the most common way is when an administrator needs to delete a VMDK but isn't quite ready to commit... so, after pressing "remove", they select "remove from virtual machine" rather than "remove and delete files".  The intent is always good: "I want to be able to restore this if it turns out to be necessary... so I'll just come back and delete it next week if nothing breaks!"  But, of course, next week something else happens and the file doesn't get deleted.  After a little while, it turns into "which files did I rename, again?" and nothing seems to happen. So, how do you deal with these orphaned files?  Well, a few years ago I found a scrip...
Read more