VCF SDDC Manager: Identity Internal Server Error

I was recently working on our lab environment at work that runs VCF 5.2.1. This “lab” is the one used in the VMware Livefire classes so it is quite similar to those used in the VMware Explore Hands On Labs. In any case, I had attempted to use Lifecycle Manager to start one of the VMs in the environment and it immediatel failed due to a bad password on vCenter. I did a RETRY on the Request and manually specified the password to use. That immediately went through to success. Since LCM was deployed via VCF’s SDDC Manager, I decided to power that up and attempt to login to it. NOTE: Since this is a lab, we don’t always power on SDDC Manager.

Symptom: Upon login to SDDC Manager in VCF 5.2.1, I received the following error message:

1{"message":"Identity Internal Server Error","code":"IDENTITY_INTERNAL_SERVER_ERROR","status":500}

At first, I thought maybe this was some sort of bad cache, so I reset the url and tried again, this time it almost looked like it was working, but displayed this:

When I clicked on the “Proceed to SDDC Manager” button, I was directed back to the JSON error message!??

I wasn’t fully sure what was causing this issue because when the lab environment was captured to a template, this all worked. My attempt to search for a solution online only yielded one result, an article written in Chinese. Thanks to Google Translate, I was able to follow the guidance given there and restore my lab environment to working condition. Big Shout out to Junior Mu - thanks for [documenting the solution](https://www.cnblogs.com/juniormu/p/18777951! If it weren’t for that article, I probably would have lost a number of hours and bothered others to help troubleshoot.

For a more detailed explanation of everything, please do visit Junior’s blog and use a tranlsater if necessary. Otherwise, the instructions below are the distilled steps I took to review and fix the issue in my lab environment.

Check Password Health in SDDC Manager

Since we see that the issue has to do with Identity, it stands to reason that a logical first step would be to check password management in SDDC Manager. However, since we’re unable to login fully due to the issue, we’ll need to do this via the command line.

1. Connect to your SDDC Manager via ssh: ssh vcf@sddc-manager.vcf.sddc.lab

2. Once logged in, run sos with the password-health switch: sudo /opt/vmware/sddc-support/sos --password-health --force

   My output:

   

   As we can see from the screenshot, SDDC Manager is unable to connect to the vCenter server: “Failed to get details”

Check the logs in SDDC Manager

HINT: You can find more detailed error in the following log files:

1/var/log/vmware/vcf/domainmanager/domainmanager.log
2/var/log/vmware/vcf/operationsmanager/operationsmanager.log

For example:

1cat /var/log/vmware/vcf/domainmanager/domainmanager.log | grep credentials
2cat /var/log/vmware/vcf/operationsmanager/operationsmanager.log | grep credentials

Showed:

1Cannot complete login due to incorrect credentials: vcenter-mgmt.vcf.sddc.lab, svc-sddc-manager-vcenter-mgmt@vsphere.local.

Based on the results above, it appears that there is a problem with the svc-sddc-manager-vcenter-mgmt@vsphere.local account that SDDC Manager is attempting to use in order to log in to vCenter with.

Use the API to get the service credential password

1. From your ssh session on the SDDC Manager, issue the following command to get a token (Replace the username and password for valid values in your own environment). The command below will get an access token from SDDC Manager so we can perform additional requests against the API.

1TOKEN=$(curl -d '{"username" : "administrator@vsphere.local", "password" : "VMware123!VMware123!"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')

1. Verify that you received a token by running the following:

echo $TOKEN

This should display a very large block of text - that is your token. You may proceed. If it does not, then you’ll need to try just the curl part of the command without the | jq …. bit, in order to see what the output is and further troubleshoot.

3. Now that you have a valid token to work with the API, run the following command to get the password of the service account:

1curl -k -X GET -H "Authorization: Bearer "$TOKEN"" --insecure 'https://localhost/v1/system/credentials/service' | jq

If you want to get fancy and ONLY return the password that matches the username of svc-sddc-manager-vcenter-mgmt@vsphere.local , we can get fancy with jq and only select the array element with the matching username and then only output the value of the .secret element:

1curl -k -X GET -H "Authorization: Bearer "$TOKEN"" --insecure 'https://localhost/v1/system/credentials/service' | jq '.[] | select(.username == "svc-sddc-manager-vcenter-mgmt@vsphere.local") | .secret'

That will return the password enclosed in quotes.

Example output of the commands above:

Update Service Account Password in vCenter

Once you have retrieved the current password that SDDC Manager has specified for the vcenter service account, you can update the password in vCenter.

1. Log in to your vCenter server with an adminstrative account. For example: administrator@vsphere.local
2. Click on the Hamburger Menut to the left of “vSphere Client” and select Administration
3. Once you’re on the Administration page, scroll down the left pane until you find the “Single Sign On” section
4. Click Users and Groups
5. In the right pane, under Users, click the dropdown next to Domain and select vsphere.local
6. Select the service account user. In my case it is svc-sddc-manager-vcenter-mgmt@vsphere.local
7. Click EDIT

NOTE: The password shown is in an isolated learning environment based on VMware Holodeck, so having it on display here is of no concern.

8. Paste in the password you retrieved from the API
9. Save

Verify

At this stage, you should re-run the sos command with the –password-health switch as done earlier.

Additionally, attempt to log in to SDDC Manager via the web UI and re-check the password management rotations:

If you’ve experienced the issue that brought you to this article, then it is likely your environment was off for some time and some passwords may have expired and/or not been rotated proplery.

NOTE: You may even need to reboot SDDC Manager and/or vCenter

Hopefully this helps! Even if it doesn’t, this article serves as a reminder for me if/when I stumble across the issue again!