Troubleshooting

SDDC-Manager Doesn’t want to start or retry an update (VCF 3.10)

SDDC-Manager Doesn’t want to start or retry an update (VCF 3.10)

I had a weird problem, where my SDDC-Manager of my VCF environment didn’t want to start any update. I had this in 2 different scenario’s. In one case I had a failed ESXi upgrade according to the SDDC-Manager. Checking it later on, the upgrade was fine, but the SDDC-Manager still marked the host in an ...read more → ...
vCF Upgrade 3.9.1. to 3.10 – vCenter Upgrade errors & issues

vCF Upgrade 3.9.1. to 3.10 – vCenter Upgrade errors & issues

Intro During the upgrade of the vCenters of one of my customers, I stumbled several times upon the issue down below. It suggest to check the logs, and to check if the ‘service-control –start applmgmt is running. On both machines the service applmgmt is running. I also checked if SSH was on, which was the ...read more → ...
How to fix Expired Password issues for vCenters & PSCs (Also for vCF environments)

How to fix Expired Password issues for vCenters & PSCs (Also for vCF environments)

So a frequently upcomming issue at customers, is the password expiration of the root password. This is set by default for 90 days, and after 90 days, well people figure out that they cannot login. To solve this, simply make a SSH connection towards the vCenter or PSC. To access shell with root type: shell ...read more → ...
How to get fully (root) access to NSX-v appliances (Manager & Edge) & NSX Intelligence

How to get fully (root) access to NSX-v appliances (Manager & Edge) & NSX Intelligence

If you have ever tried to troubleshoot an NSX-v Management Appliance or Edge, you probably noticed that you are quite limited in the execution of your controls. That’s because in NSX-v you are standardly limited with most of the time only esxcli, even when you are logged in as admin. To get past that you ...read more → ...
Check if the Replication is working between PSCs

Check if the Replication is working between PSCs

For a customer I was replacing the SSO/STS certificates, after the change wanted to make sure if the PSCs were still replicating the information between each other. To simply check this Type the command: ./vdcrepadmin -f showpartnerstatus -h localhost -u administrator You can see who the partner is of the PSC, and how much changes ...read more → ...
vRLI Webgui doesn’t start up: Config Folder Lost & “Keystore was tampered with or Password was incorrect”

vRLI Webgui doesn’t start up: Config Folder Lost & “Keystore was tampered with or Password was incorrect”

Intro One of my customers had problems with their vRLI environment, the most obvious one was that the webgui wasn’t available or even started. The following post is a more general approach of how I troubleshoot vRLI. In this case I had to apply 2 workarounds / solutions to make it work again. In this ...read more → ...
Check & Renew the expiration of the Machine Certificate of the PSCs & vCenter (vCenter 6.5/6.7)

Check & Renew the expiration of the Machine Certificate of the PSCs & vCenter (vCenter 6.5/6.7)

Intro At a customer we got the notification in the vCenter that the certificate was almost expiring. This was about the SSO/STS certificate. Later we found out that also the machine certificates were expiring on each PSC and vCenter that were deployed 2 years earlier. This post will focus on the renewal of the Machine ...read more → ...
NSX-T Passwords Expired for default accounts

NSX-T Passwords Expired for default accounts

A colleague of my, had some troubles making an API call to the NSX-T Managers in a particular site. Looking closer into vRA we saw the following message: So we decided to log into one of the NSX-T managers, and we directly saw the following: Since the password was expired, we were asked to change ...read more → ...
Can’t access the NSX-T managers with SSH in vCF?

Can’t access the NSX-T managers with SSH in vCF?

When trying to connect to a NSX-T manager node with SSH, it is possible that you get this message. That is because during the deployment of NSX-T, the SDDC-Manager doesn’t turn on SSH for the VMs. This can simply be changed by login in to the webconsole of the NSX-T manager, and turning on SSH ...read more → ...
Renewing the default STS (SSO) Certificate of vCenter/PSC in a vCF Environment

Renewing the default STS (SSO) Certificate of vCenter/PSC in a vCF Environment

At a customer we got the notification in the vCenter that the certificate was almost expiring. However, the vCenter with version 6.7 was part of a vCF Deployment. Almost all the certificates are being managed by vCF, so when we checked the status, it didn’t seem that anything was expiring soon. However, the SSO STS ...read more → ...
Warning: SSL Certificate almost Expired

Warning: SSL Certificate almost Expired

SSL Certificate almost Expired At a customer we got the notification in the vCenter that the certificate was almost expiring. When I checked the HTML client, I couldn’t find the SSL certificate that the warning was talking about. Looking deeper, I noticed that the SSO STS Certificate, which are standard issued by VMware and being ...read more → ...
Error when trying to access VMware PSC with WinSCP (6.7)

Error when trying to access VMware PSC with WinSCP (6.7)

For a customer, I needed to upload a script towards the PSC, and my favourite tool for that is WinSCP. However, when trying to access, I got the following Error. The reason for this is that the default shell within the PSC, doesn’t allow access from the WinSCP. So we need to change default shell ...read more → ...
vCF 3.x – SDDC-Manager Precheck Fails: LCM Directories Permission Check

vCF 3.x – SDDC-Manager Precheck Fails: LCM Directories Permission Check

Problem Summary For a customer, I was doing the prechecks for his VCF environments, since we wanted to upgrade the environments from 3.9.1. to 3.10. Unfortunately the following error appeared on all 3 environments.   Error Description: LCM does not have read/write permission to /opt/vmware/vcf/lcm/lcm-app/conf/jpa-prod.properties.bak Proposed Remediation: Make sure LCM App user (vrack) has read ...read more → ...
Deployment of a OVF / OVA is stuck with Validating…

Deployment of a OVF / OVA is stuck with Validating…

Issue At one customer I was deploying vRNI. For some reason, every time when I selected the ova file that I downloaded from VMware, it stayed stuck in the “Validating” screen. This could take a long time and would eventually end up failing. Workaround In my case the problem seemed to be that I wasn’t ...read more → ...
vCF 3.x – SDDC Manager UI issue after upgrade to 3.8 – failed to poll or fetch info

vCF 3.x – SDDC Manager UI issue after upgrade to 3.8 – failed to poll or fetch info

Intro So I made a separate post here, that lists up all the different reasons why this issue could arise. However, the issue described here is more and less a known issue for the upgrade of the SDDC Manager from version 3.7.2. to version 3.8. This didn’t happen at our first site were the SDDC ...read more → ...
vCF 3.x – SDDC Manager fails to poll or fetch info within the webUI

vCF 3.x – SDDC Manager fails to poll or fetch info within the webUI

Intro During the different upgrades of vCF, one issues came several times back within the SDDC Manager. Which is that the SDDC Manager is unable to show any kind of information or data within the web UI. General Reason for failing So there can be several different reasons and scenario’s why this is happening. However, ...read more → ...
vCF 3.x – vRealize Suite Lifecycle Manager Environment Page is Blank after Upgrade

vCF 3.x – vRealize Suite Lifecycle Manager Environment Page is Blank after Upgrade

Issue For my customer I upgraded the vRSLCM from version 1.2 to version 2.0 which is in the BOM of vCF 3.5. During the upgrade I had several issues, but we solved it. However, after a successful upgrade, the home Page with all the environments was blank. At first I was afraid that this was ...read more → ...
vCF 3.x – Upgrade Precheck fails at ESXi hosts

vCF 3.x – Upgrade Precheck fails at ESXi hosts

Problem Recap When starting a vCF upgrade, it is always wise to start the precheck. At one of our customers, the precheck failes quite often on the ESXi hosts. The most common reason for this was that the customer didn’t have a persistent scratch partition, and with that, also not a ”scratch/downloads” folder. Another reason ...read more → ...
vCF 3.x – Web GUI looks weird after RPM upgrade of SDDC-Manager-UI-app.service

vCF 3.x – Web GUI looks weird after RPM upgrade of SDDC-Manager-UI-app.service

Intro In one of my posts I explained that we had a problem with the upgrade of the SDDC Manager, especially with the SDDC-Manager-UI component. It once again failed and I needed to do an upgrade of the RPM manually. But what I noticed, was that when ever I upgraded the RPM of SDDC Manager ...read more → ...
vCF 3.x – vRSLCM 2.x upgrade fails after deployment

vCF 3.x – vRSLCM 2.x upgrade fails after deployment

Intro At one of my customers, we had a problem that the vRealize Suite Lifecycle Manager (vRSLCM) didn’t want to upgrade. We already faced quite some problems with the vCF upgrades, so this one was a nice extra present. This happened when we were upgrading from vCF 3.0.1.1 towards 3.5. In this upgrade we had ...read more → ...
vCF 3.x – SDDC Manager Upgrade fails at SDDC-Manager-UI Component

vCF 3.x – SDDC Manager Upgrade fails at SDDC-Manager-UI Component

Problem Recap At a customer we had an issue that came back every SDDC-Manager upgrade within the 3.x versions. The SDDC-Manager doesn’t seem to start the upgrade of the SDDC-Manager-UI.app service. In the logs there is almost nothing to find of why this isn’t starting. When looking really closely, we see that the RPM is ...read more → ...
vCF 3.x – No bundles available in the SDDC Manager

vCF 3.x – No bundles available in the SDDC Manager

Problem Recap After a successful upgrade of the SDDC Manager to version 3.7.1, the UI of the SDDC Manager doesn’t show me any new bundles that can be downloaded. This issue is commonly known for version 3.7.1. and that is because the SDDC Manager needs 2 updates before it can continue with the other components. ...read more → ...
Troubleshooting: Cannot Add host “Authenticity of the host’s SSL certificate is not verified”.

Troubleshooting: Cannot Add host “Authenticity of the host’s SSL certificate is not verified”.

Intro One of my customers wanted to change the hostname of the ESXi host. In this case, the ESXi hostname stayed the same, but it needed to become part of  a subdomain. Changing an ESXi hostname This can normally be easily done in the following way: Migrate VMs, offline VMs and templates of the host. ...read more → ...
VUM Error: Host cannot download files from VMware vSphere Update Manager patch store

VUM Error: Host cannot download files from VMware vSphere Update Manager patch store

Setup: vCSA 6.5 with embedded VUM Intro: At one of my customers, I had a weird problem that one of the hosts couldn’t updated through VUM. Time to investigate. Error: “Host cannot download files from VMware vSphere Update Manager patch store.  Check the network connectivity and firewall setup, and check esxupdate logs for details.” When ...read more → ...
vCenter doesn’t start up – Failed to start File System Check on /dev/dis..

vCenter doesn’t start up – Failed to start File System Check on /dev/dis..

A customer of mine had some problems with a vCenter (vCSA 6.5) that didn’t want to boot up anymore. First thing I checked was if I could log in on the vCenter Appliance Management Interface (VAMI). Unfortunately both the webclient as VAMI was unresponsive. This was also the case with SSH. So the last thing ...read more → ...
TIQR (SURFnet) stopped working after Android Update 9 (Android Pie)

TIQR (SURFnet) stopped working after Android Update 9 (Android Pie)

For one of my customers, I use TIQR to login remotely. However, after an Android Update towards version 9, the application stopped working. In the beginning it looked like the application didn’t have any problem, since I still would get a push notification that would ask for my code. However, every time I typed down ...read more → ...
VM has No Network Connection, Windows 7 – VMware Fusion 10.1 (Workstation)

VM has No Network Connection, Windows 7 – VMware Fusion 10.1 (Workstation)

Recently I had a weird issue that a Windows 7 virtual machine on VMware Fusion (10.1) wasn’t able to make any network connections. After doing all the basic checks (NAT configured, Windows Firewall, Network adapter connected in VM) I didn’t had a clue what was causing the issue. Still, the problem looked like an issue ...read more → ...
vSphere Troubleshooting: Hosts Disconnects  “Agent unable to save configuration to disk: Error syncing firmware configuration: vim.fault.TooManyWrites”

vSphere Troubleshooting: Hosts Disconnects “Agent unable to save configuration to disk: Error syncing firmware configuration: vim.fault.TooManyWrites”

During a deployment of several ESXi hosts at a customer, I noticed that the new hosts were disconnecting every few minutes. When I checked the event error, it showed a lot of different errors and events, mostly about disconnection errors and connection failures. Which I found weird. Looking closer, the error message reads: “agent unable ...read more → ...
Troubleshooting: vSphere Sessions are constantly logging off –  Connection Timeout

Troubleshooting: vSphere Sessions are constantly logging off – Connection Timeout

I recently had a problem that one of my vCenters (vCSA 6.5) was constantly logging off any kind of user after a few minutes. Which would end up with the following error output: “Your vsphere Client session is no longer authenticated Please login again”. This even happend after 2 minutes whenever I was logged in. ...read more → ...
Retrieving License Keys and other Valuable information from HPE Servers

Retrieving License Keys and other Valuable information from HPE Servers

Sometimes, you want to have more info then the GUI shows you, or you want to create or change configuration on loads of different servers, and clicking is not the fastest way. Command line and scripting tools always have been a nice way to automate things or to retrieve valuable information that can’t be found ...read more → ...
vCSA6.5 Installation Stuck at 80% and some other fun errors during Deployment.

vCSA6.5 Installation Stuck at 80% and some other fun errors during Deployment.

if you came across this post then you’ll probably had the problem that the vCenter installation became stuck around the 80%, with the text “Installing RPM This may take several minutes”. Unfortunately, this doesn’t take minutes but several hours until it figures out that it got an issue and it just stops. Like this screen: ...read more → ...
Crashed Synology Volume, and how to Restore (DS415 Play)

Crashed Synology Volume, and how to Restore (DS415 Play)

Intro As an IT guy, I have started to get a little bit of my own personal lab, and when sometimes something breaks, I always try to see it as a fun way to gain some experience and put on my troubleshooting hat. The fun stops however, when you know that you might possibly lost ...read more → ...