Intro
We’ve viewed the different tools that help to manage a HPE stack and if and how we could integrate VUM with this. We also discussed that consistency with virtualization & infrastructure is one of the most important things for having reliable operations, and that you need some sort of reporting tool that can give you a clear and trustworthy overview. With the scripts and knowledge from the previous post, it is time to define a form of LCM Plan. That’s why this post will focus on how we can update and manage a VMware vSphere environment on a HPE stack. This (global) plan will focus itself purely on upgrading the firmware, ESXi host & Drivers, and to keep it compatible and consistent, since they are closely related and therefor important.
This LCM plan is not focused on upgrading the complete VMware Stack. For the complete VMware stack, I’m writing a different blogpost series called “Creating a Roadmap for vSphere Upgrades”.
Components to be Updated & Upgraded
Simply said, while maintaining a VMware & HPE Hardware environment, the following should be upgraded & checked:
- HPE Stack (Firmware)
- HP OneView
- 3PAR
- Blade Enclosures
- Onboard Admin of (Blade) Enclosures
- Virtual Connects
- Blades
- VMware vCenter
- ESXi updates
- ESXi drivers
Previous blogposts explain on which criteria we selected our tools and what the pros vs cons were. In our verdict, we saw that at this moment of writing (15-05-2019), we currently prefer HP OneView for pushing out Firmware updates, and let VMware VUM take care of the ESXi updates & Drivers. We also saw that there was a lack of reporting within the tools. The scripts that were created to solve this issue are taken in this plan as well.
* Note: We used & tested the tools on a VMware 6.5 vSphere environment with “HPE Proliant BL460c G9” servers & “3PAR” storage. The Bl460c Servers are part of an enclosure (Converged Infrastructure) that has its own Onboard Admin and uses Virtual Connects to manage the network connections. This procedure was created for that type of environment. However it will probably also apply for most other HPE environments. If you are not using a CI/HCI setup, this will probably become easier, since there are lesser components to worry about.
Plan & Order of the Upgrade
The order of the upgrade is very important. In general, it comes down to the following.
1.0 Preparation
— 1.1 Start with a consistency check by running the scripts from the previous posts
- TheVibReporter Scripts will give you insight of all the vibs that are installed within the vSphere environment and if there are any indifferences.
- For HPE it is also possible to create a script that will check all the firmware versions within the environment. Contact HPE support or a representative to assist you with this. Only then you know for sure which jumps you have to make.
- Running RVTools before you start also never hurts 😉 (love that tool).
— 1.2 Check the HCL of VMware
- Check the VMware site to see if the new HPE firmware is supported for the future VMware version. Although there are official HPE SPPs, no automatic check is done if the underlying Operating System also supports the firmware, in this case ESXi. A manual check is therefore necessary.
— 1.3 Create Backups & Migrate workloads and /or schedule downtime
— 1.4 Download the SPP of HPE
2.0 Upgrade of Hardware Stack (HPE)
This will be the order that we used for our Converged Blade infrastructure with 3PAR Storage. This was the order that we needed to use when we upgraded from a SPP of 2015, up till 2018-Nov. During the different upgrades, this was the general order that always came back.
— 2.1 Upgrade HP OneView
- Check if an upgrade is required. Most of the time this has to be done first but always check if the OneView upgrade also support the used & future firmware versions of your managed hardware.
— 2.2 Upgrade the firmware of 3PAR
- Check if this is needed and if this involves downtime. There is not always a downtime or upgrade needed for these cases.
- Sometimes upgrades can bring new features. With our 3PAR we could go to a newer dedup version (3). Which together with the newer VMFS6 filesystem, made quite a positive impact on storage usage. More importantly, is that an upgrade of features sometimes requires some drastical actions as well before it is fully implemented. In our case a new CPG had to be created and all the LUNs had to be moved over, which needs quite some time and planning.
— 2.3 Upgrade the firmware of the Onboard Admin of the Enclosure (before the blades)
— 2.4 Upgrade the firmware of the Virtual Connect (before the blades)
- Make sure that there is a redundant network link or that downtime is calculated. Also, a nice thought to inform your network team 😉.
— 2.5 Upgrade the firmware of the Blades
- With HPE OneView, we used “Server Profiles” & “Server Profile Templates”, which made this process a little easier and at the same time also more difficult.
- Check The lessons learned section, since a lot of strange things happened several times during this phase.
— 2.6 Check if Upgrades are Successful & Consistent (Firmware)
- Check if all the components are upgraded correctly, and if it is consistent across the environment.
- Ask HPE Support verifying this, since this is almost impossible to do with the current and standard HPE tools.
3.0 VMware vCenter & ESXi Upgrade
Once you upgraded all the HPE Hardware and checked if everything is compatible and up and running, the next step will be upgrading the VMware Stack. Now do note that a vSphere environment can consist of a lot more components, which we do not focus on in this post. However, in almost every case, (hardware) firmware versions & ESXi version + Drivers, are tightly linked and connected and therefor follow each other closely with compatibility. Other tools, appliances & components of VMware, may have a soft link with the hardware & firmware version. Still, these components are most of the time hardware & firmware independent, where an ESXi host is definitely not. Even if you would do a complete VMware vSphere Upgrade with other components (NSX, vSAN, vRA, vROps, etc.) the order of Upgrade for this part will almost always be the same. It is most of the time not the upgrade of the ESXi host or firmware/hardware that should be checked for compatibility with other VMware components (except for vSAN), but the upgrade of the vCenter version. The thing is, you can only upgrade your ESXi host if your vCenter is on a higher or similar supported level. So definitely check out before you begin if you have to upgrade your vCenter and if this is possible with other components. With that being said, the general order of a VMware Upgrade in this case would be:
— 3.1 vCenter Server
- Check if other VMware components are compatible with this upgrade.
- Check my other BlogPost Series “Creating a Roadmap for vSphere upgrades” for guidance on this topic.
— 3.2 ESXi (OS) Updates
- Check the HCL if the firmware version is supported with ESXi version
- We use VUM for these updates & upgrades
— 3.3 ESXi drivers
- If needed and thoroughly checked, you can upgrade the drivers as last. This can be done by adding the HPE Repositories into VUM.
— 3.4 Check if Upgrades are Successful & Consistent (Vibs)
- Check if all the ESXi hosts are upgraded correctly, and if the installed vibs are consistent across the environment.
- TheVibReporter Scripts can help you with this.
Learned Lessons during the Upgrade
During the upgrade various issues & challenges occurred several times. So a quick list of the things that I thought are quite important to know.
- Upgrade Blade fails
- This was mainly because the ILOs had not been upgraded and were too far behind with their version. By first upgrading the ILO, the upgrade often worked again afterwards.
- SD cards were lost
- This problem was also often solved by the upgrading the ILOs. However, it also happened quite some time that a SD card still had to be replaced. Before starting the upgrade, make sure that you have some Spare SD cards in advance.
- ESXi hosts do not find any data stores or network connectivity
- This actually always had to do with the server profile, which was not properly loaded from HP OneView. In our setup, the blades didn’t make use of their original mac addresses & WWNs, but the ones that were defined in their profile. It is therefore very important that this is correctly loaded or reapplied. When a Server Profile was not (correctly) loaded, the blade got its original mac addresses & WWNs. The original mac adresses &WWN were not known on the Virtual Connect & 3PAR, which means that the ESXi host could not find any network and / or data stores.
- This can be solved by reapplying the profile again. If that does not work, an ILO reset must be performed, and the profile must be reapplied again. If that also does not work, removing and re-adding the blade in the OneView Server appears to be the best and last resort.
- It is also important to know that you shouldn’t reapply too many settings at once in the HP OneView Gui. It seems that when all settings from the “Server Profile” with HP OneView are pushed toward the blade together with the upgrade, that not everything will be loaded. The speed of the reboots seems to also have an effect on this.
- Not all Firmware is being upgraded or inconsistent while saying everything is OK
- This happened quite often, so we have developed a script that performs a reliable check. This script is not publicly available, so ask HPE for support on this issue.
- Roll back a VMware ESXi update
- For testing purposes, we we wanted to know the effect of several updates or setups. However, after the tests, we wanted to go back to the original version afterwards. It is good to know that an ESXi host has an alternative bootbank that is created after 1 VUM update, to which the previous installation will be moved. This alternative bootbank can be opened from the ESXi host. You do this by restarting an ESXi host and by pressing “Shift-R” on your keyboard during the Hypervisor Boot Phase. This opens the “VMware Hypervisor Recovery” menu, where you can select the old installation. It only saves 1 alternative installation. So, if you upgrade the host a 2th time from VUM, the host will only save the last known configuration on the alternative bootbank. I made a separate blogpost about this, so check it here, if you are interested.
Like I said earlier, these steps & plan are mostly based on the VMware & HPE setup discussed earlier.
Still, I do think that this give you some great guidance & head start when upgrading a virtualized hardware on HPE hardware.
What do you think? Does this post help you, or does it feel incomplete for you.
Let me know in the comments below, and what you would love to see.
Samir
Samir is the author of vSAM.Pro & a Life enthusiast who works as a consultant in the field of IT. With a great passion for Tech & Personal Development, he loves to help people with their problems, but also inspire them with a positive outlook on life.
Besides that, he is also a big Sport & Music junky that loves to spend a big chunk of his time on producing music or physically stretching himself.
