Updating iLO on a BL460c ESXi Blade
Several of my customers use HP C7000 blade chassis for their ESXi hosts. One of them asked me to help them update the firmware in their enclosures. There are basically three different pieces of the blade chassis; each one needs updating. The blades themselves need updating, the Interconnect Modules (basically, embedded switches) need updating and the Onboard Administrator Modules need updating. There are different and varied steps for updating each of those components. The blade is updated by bootstrapping with an SPP (Service Pack for Proliant) ISO. The OA is updated through its own web interface by uploading a specific .bin file. The Virtual Connect (which is the centralized management for all of the Interconnect Modules) is updated through its own special command line utility by uploading its own specialized .bin file.
All of that is a little awkward, but it’s not really that big of a deal. We did run into one spot of difficulty, as one of the BL460c ESXi hosts was reporting that it had lost access to its SD card. A bit of research pulled up an interesting article from HP about a blade firmware update that would resolve the issue. Seeing as how we were doing an update cycle anyway, this seemed like it would be a nice easy fix! Of course, since I’m writing about it, you can probably guess that it didn’t work out quite so easily.
We specifically had to update the iLO firmware on the malfunctioning blade (and eventually, all of the blades in order to head off this issue). Applying blade level updates, including iLO, is easily done by bootstrapping the system with an HP SPP disk. The latest SPP disk (at the time of writing) is from September 2014 and updates the iLO firmware to version 2.00; the version that we needed to apply is version 2.03. So, when you click on the download link next to the 2.03 firmware update, what do you get? You get an .scexe file (after navigating the system and selecting ESX 5 as the Operating System). Drilling down into the installation instructions reveals them to be functionally useless (at least, if you’re trying to apply the update to a system that can’t boot into an OS). When I read those instructions, I basically understood that I had to generate my own custom SPP ISO, with the iLO firmware .scexe baked into it. Some more googling brought me to a slightly more verbose description of how to create a custom SPP, but I wasn’t able to get that working (and after waiting through the ISO creation process... and the bootstrapping process, only to have it fail without an error message a few times… well, I was pretty frustrated with the whole process).
So, I got to thinking. If you log into the iLO of the blade (either directly or through the chassis’ OA proxy), there’s a firmware update section (under Administration -> Firmware). This section has an upload button, but no clear indication of what file you’re actually supposed to upload (I tried the .scexe and it returned an error, stating that it was not a valid file). I read through the installation instructions for some other OSes and saw that the .scexe file is really just a container file that has the needed .bin inside of it. Of course, I had no idea about how to get into this particular file, as it was built to run on ESXi (and I didn’t want to test this theory on another, working host). So, I went back to the HP download site and downloaded the Windows Server 2012 iLO firmware package (after all, firmware is firmware, regardless of the OS that’s running above it). I was able to run this executable on my desktop (pressing “Extract” rather than “Install” when prompted) and, there in my directory was sitting the .bin file that I needed!
So, I went into the blade’s iLO, went to Administration > Firmware and then browsed to the .bin file that I had extracted. I pressed the upload button and watched the system go through its error checks and… success! The iLO system took a bit longer to reboot than I expected (it came back to the login prompt almost instantly, but things were sluggish and occasionally 404’ed for a couple of minutes after the update). After I gave it a few minutes to complete its process, everything seemed to be nice and stable and, when I restarted my ESXi installation, the SD card was ready and available for me to install upon.
All of that is a little awkward, but it’s not really that big of a deal. We did run into one spot of difficulty, as one of the BL460c ESXi hosts was reporting that it had lost access to its SD card. A bit of research pulled up an interesting article from HP about a blade firmware update that would resolve the issue. Seeing as how we were doing an update cycle anyway, this seemed like it would be a nice easy fix! Of course, since I’m writing about it, you can probably guess that it didn’t work out quite so easily.
We specifically had to update the iLO firmware on the malfunctioning blade (and eventually, all of the blades in order to head off this issue). Applying blade level updates, including iLO, is easily done by bootstrapping the system with an HP SPP disk. The latest SPP disk (at the time of writing) is from September 2014 and updates the iLO firmware to version 2.00; the version that we needed to apply is version 2.03. So, when you click on the download link next to the 2.03 firmware update, what do you get? You get an .scexe file (after navigating the system and selecting ESX 5 as the Operating System). Drilling down into the installation instructions reveals them to be functionally useless (at least, if you’re trying to apply the update to a system that can’t boot into an OS). When I read those instructions, I basically understood that I had to generate my own custom SPP ISO, with the iLO firmware .scexe baked into it. Some more googling brought me to a slightly more verbose description of how to create a custom SPP, but I wasn’t able to get that working (and after waiting through the ISO creation process... and the bootstrapping process, only to have it fail without an error message a few times… well, I was pretty frustrated with the whole process).
So, I got to thinking. If you log into the iLO of the blade (either directly or through the chassis’ OA proxy), there’s a firmware update section (under Administration -> Firmware). This section has an upload button, but no clear indication of what file you’re actually supposed to upload (I tried the .scexe and it returned an error, stating that it was not a valid file). I read through the installation instructions for some other OSes and saw that the .scexe file is really just a container file that has the needed .bin inside of it. Of course, I had no idea about how to get into this particular file, as it was built to run on ESXi (and I didn’t want to test this theory on another, working host). So, I went back to the HP download site and downloaded the Windows Server 2012 iLO firmware package (after all, firmware is firmware, regardless of the OS that’s running above it). I was able to run this executable on my desktop (pressing “Extract” rather than “Install” when prompted) and, there in my directory was sitting the .bin file that I needed!
So, I went into the blade’s iLO, went to Administration > Firmware and then browsed to the .bin file that I had extracted. I pressed the upload button and watched the system go through its error checks and… success! The iLO system took a bit longer to reboot than I expected (it came back to the login prompt almost instantly, but things were sluggish and occasionally 404’ed for a couple of minutes after the update). After I gave it a few minutes to complete its process, everything seemed to be nice and stable and, when I restarted my ESXi installation, the SD card was ready and available for me to install upon.
Jason have you had a problem with connecting to the irc using the OA
ReplyDeleteI'm not familiar with "irc" in this context.
Deleteit is in the OA itself
Deletesecond how do you update the ilo for the blades using the OA
ReplyDeleteUnder "Administration" click on the "Firmware" section. You'll need the correct .bin file for that to work though.
DeleteJason, I have C7000 OA at 3.21. Recently the OA encounters an SSL_ERROR_BAD_MAC_ALERT when trying to get to any of the 16 ILOs [I run wireshark to see the cert presented by the ILO]. This occurs for any browser, PC, and even access using direct to ILO [without OA]. Chatting with HP, they say to upgrade OA, but the upgrade won't take. Since this occurs with all 16 blades, it must have something to do with OA? The only clue I have is the date of 11-27-2016 in the OA cert admin -when this started. But what would this have to do with it? Can't seem to get anywhere.... thanks for any help. John
ReplyDeleteYeah, it sounds like an OA issue to me because of the scope, but I don't have any more insight than that. That said, even if the OA is down, I think that you should be able to connect to the iLO directly (although it'll still route that traffic through the OA's network adapter), so maybe not. Have you tried doing a hard reset on the iLO on one of the blades to see if that clears it up? I wish I had a better idea for you.
DeleteThanks for your response Jason! OA is not down, I can still access it just fine with any browser. Just the access to ILO via this [or directly vioa browsr] gives me the bad cert warning. I still did putty directly to OA, logged on and then did a "CONNECT SERVER 1 [e.g.]" and then cd /map1 and then "reset" the ILO. [can't remember where I got all that....] Nothing changed. Is that a hard reset on the ILO? The cmd did take and I was able to come back in. Don't see much there via help to get anywhere. JOhn
ReplyDeleteSorry that I haven't seen this one myself, but have you seen this article https://www.hahosting.info/2015/07/hp-ilo3-ssl_error_bad_mac_alert-error-when-connecting/? That makes sense to me as a potential source of the issue. Also, what did you mean when you said that the iLO upgrade won't take? I',m guessing that you can't you get into the Web Administration for the blade and attempt to update the firmware directly via the .bin file as I described; here's an article that describes performing that update via SSH: http://techmolecules.blogspot.com/2014/08/four-ways-to-update-hp-ilo-firmware.html
DeleteHmmm. Sounds possible. Thot I went there once-suspected something like that, start to forget all these steps. But my browser worked till Dec 2016. But I had not encountered that link. Thanks!! I will go back on that path for sure. I could not update OA 3.21 all the way to 460... I suspect I need an intermediate step [OA told me he couldn't update to that firmware]. Once I get OA updated, will try to update ILO.... too much fun. Thanks again!
ReplyDeleteJason,
ReplyDeleteI managed to upgrade OA to 3.6 (same as my other C7000). Still no go tho accessing the ILO.
I then managed to turn off TLS 1.2 in IE, and was able to "get" to the ILO. Once there I got a warning that the digital signature on "com.hp.ilo2.intgapp.intg.app" was signed with a trusted cert, but that it had expired..... So I guess an ILO update is in order...
BUT If I understand correctly, I will have to reboot all my blades with SPP! Not sure I want to take all those servers down real soon! Lots of stuff running on them :-(
Of course I have to use an outdated java to run all this.
So I guess I am back in business. Add to your own knowledge base I guess. I appreciate your help!!!! Take care. John
Glad that I could help. I believe that you can update iLO through the Web Administration page without impacting the actual server by uploading the iLO bin directly.
Delete