HP c7000 Chassis Administration Tips and Tricks
Several of my customers use HP C7000 Blade Chassis for their ESXi hosts. I've picked up a few tips and tricks for working with that chassis over the years, so I figured that I'd post them here.
The Virtual Connect (the blade chassis's networking component) has a feature that can prevent pause frames from flooding a network by disconnecting a blade that is sending an excessive number of them. Unfortunately, every now and then, it detects an ESXi host's uplink as sending such a number of pause frames and so disconnects that network adapter. Fortunately, it's really easy to allow traffic to flow through that port once again. Just SSH into the Virtual Connect (you can get the address by looking at the "Virtual Connect Manager" link in the Onboard Administrator interface. Once you're connected, use the show port-protect command to see if there are any ports that are in a blocking state. If so, you can use the reset port-protect command to reset the pause flood blocking (it's a global thing), and your link should come right back up.
Another issue that I see occasionally is when the server loses its access to its SD card. Disconnecting power from the blade and then powering it back on will frequently restore access, but that's not easy to do if the blade isn't physically near you. Fortunately, there is a way to remotely trigger the e-fuse on the blade and cause such a hard reset. I shut down the blade first, then SSH into the Onboard Administrator. Once I'm signed in, I used show server status X where X is the blade slot that I'm interested in. I verify that the power is off for that blade, just to double check that I'm really working on the server that I think that I'm working on. Once I'm sure that X is really the blade that I want, I use reset server X to trip the e-fuse. Within a minute, I see the blade drop out of the OA and then return, and it will usually bring back the SD card with it.
I recently had to do some work on a blade while troubleshooting a suspected hardware failure. One of the troubleshooting steps was to clear the NVRAM on the blade, and HP helpfully sent me a procedure. There's a little switch on the motherboard and, for this model/generation combo (BL460 Gen 8), little is an understatement. If you need to do this, you'd be well served by bringing a long needle with you, so that you can toggle only the switch that you want to toggle. In a pinch, I ended up using one tine of the star allen wrench that HP embeds in the blades. It wasn't easy, but it wasn't too bad either. Once that switch was toggled, I had to turn on the server and let it do its thing, then turn it back off again.
In order to do that process, I had to look at the console of the blade to watch the boot process. In the datacenter, I didn't have easy access to my laptop for iLo, so I learned how to use the KVM on the back of the C7000 chassis. It's not too difficult, just plug in a VGA monitor and a USB mouse/keyboard to the active OA module. If you plug into the standby module, it'll tell you, so just switch it over. Then, use the front diagnostic panel and select the "KVM Menu" option, which will black out that panel. Go back to the monitor and you should see a list of all blades installed in the system. From there, you can do power operations or select the name of the blade to open its console directly. The part that wasn't well documented is that you can get back to that menu at any time by pressing Print Screen twice in rapid succession. So, when you are ready to switch over to another blade, just press Print Screen, Print Screen, and after a couple of seconds it'll bring you back to the menu.
The Virtual Connect (the blade chassis's networking component) has a feature that can prevent pause frames from flooding a network by disconnecting a blade that is sending an excessive number of them. Unfortunately, every now and then, it detects an ESXi host's uplink as sending such a number of pause frames and so disconnects that network adapter. Fortunately, it's really easy to allow traffic to flow through that port once again. Just SSH into the Virtual Connect (you can get the address by looking at the "Virtual Connect Manager" link in the Onboard Administrator interface. Once you're connected, use the show port-protect command to see if there are any ports that are in a blocking state. If so, you can use the reset port-protect command to reset the pause flood blocking (it's a global thing), and your link should come right back up.
Another issue that I see occasionally is when the server loses its access to its SD card. Disconnecting power from the blade and then powering it back on will frequently restore access, but that's not easy to do if the blade isn't physically near you. Fortunately, there is a way to remotely trigger the e-fuse on the blade and cause such a hard reset. I shut down the blade first, then SSH into the Onboard Administrator. Once I'm signed in, I used show server status X where X is the blade slot that I'm interested in. I verify that the power is off for that blade, just to double check that I'm really working on the server that I think that I'm working on. Once I'm sure that X is really the blade that I want, I use reset server X to trip the e-fuse. Within a minute, I see the blade drop out of the OA and then return, and it will usually bring back the SD card with it.
I recently had to do some work on a blade while troubleshooting a suspected hardware failure. One of the troubleshooting steps was to clear the NVRAM on the blade, and HP helpfully sent me a procedure. There's a little switch on the motherboard and, for this model/generation combo (BL460 Gen 8), little is an understatement. If you need to do this, you'd be well served by bringing a long needle with you, so that you can toggle only the switch that you want to toggle. In a pinch, I ended up using one tine of the star allen wrench that HP embeds in the blades. It wasn't easy, but it wasn't too bad either. Once that switch was toggled, I had to turn on the server and let it do its thing, then turn it back off again.
In order to do that process, I had to look at the console of the blade to watch the boot process. In the datacenter, I didn't have easy access to my laptop for iLo, so I learned how to use the KVM on the back of the C7000 chassis. It's not too difficult, just plug in a VGA monitor and a USB mouse/keyboard to the active OA module. If you plug into the standby module, it'll tell you, so just switch it over. Then, use the front diagnostic panel and select the "KVM Menu" option, which will black out that panel. Go back to the monitor and you should see a list of all blades installed in the system. From there, you can do power operations or select the name of the blade to open its console directly. The part that wasn't well documented is that you can get back to that menu at any time by pressing Print Screen twice in rapid succession. So, when you are ready to switch over to another blade, just press Print Screen, Print Screen, and after a couple of seconds it'll bring you back to the menu.
Comments
Post a Comment
Sorry guys, I've been getting a lot of spam recently, so I've had to turn on comment moderation. I'll do my best to moderate them swiftly after they're submitted,