This one will be short I promise. After installing Lync Server 2013 CU4 (January 2014), everything worked fine on the Front Ends, but ran into a rather weird issue with an Edge Server.
After installing the update, the Lync Server Access Edge service refused to start, all other services started ok. It bombed out pretty quickly with the following error…
|
1 |
Start-CsWindowsService : Service RTCSRV failed to start within the expected timeframe. |
And Event Viewer recorded these…
Event 12303, LS Server
|
1 |
The protocol stack reported a critical error: code C3E93C0D (Configuration failure prevented the server from starting up). The service has to stop. |
Event 14694, LS Protocol Stack
|
1 2 3 4 5 6 |
A serious problem related to certificates is preventing Lync Server from functioning. Unable to use the certificate configured for the external edge of the Access Edge Server. Error 0x800B0109(CERT_E_UNTRUSTEDROOT). The certificate may have been deleted or may be invalid, or permissions are not set correctly. Ensure that a valid certificate is present in the local computer certificate store. Also ensure that the server has sufficient privileges to access the store. |
Event 14397, LS Protocol Stack
|
1 2 3 |
A configured certificate could not be loaded from store. The serial number is attached for reference. Extended Error Code: 0x800B0109(CERT_E_UNTRUSTEDROOT). |
How bizarre, it had been working fine for several months, nothing was due to expire.. Quickly checking the Local Machine’s certificate store reassured me the certificate and it’s full chain were valid and trusted.
I’d just like to add, the Serial Number supplied in the Details tab of the event log didn’t match anything on the server.. (this turned into a little detour). The Serial Number in the Event Log was 5202FBA7CF1487, and the Serial Number assigned to the external certificate was 7841FC7ABF2025 (Names changed to protect the innocent)… Running the following PowerShell confirmed that the Serial Number simply doesn’t exist.
|
1 |
Get-ChildItem -Path CERT: -Recurse | FT Subject, SerialNumber | FindStr 5202FBA7CF1487 |
Wait… Any body spot it? Is it glaringly obvious? It certainly wasn’t at first… but then it hit me… It’s backwards. The serial number displayed in Event Viewer is backwards… Why would anybody be so inclined to display a descriptive error message containing a certificate Serial Number, for a certificate it was trying to identify, in reverse?!? Anyway.. so it DOES exist, and as I saw earlier it was valid and trusted, and now I know we’re talking about the same certificate.
I can’t honestly remember each thing I tried before I stumbled on the fix., here are some of the attempts that failed to do anything:
- Ensured topology and CMS were replicated, and all UpToDate. (It was, and everything was replicating fine).
- Removed, Deleted, re-imported, and re-assigned the certificate. (Deployment Wizard showed me the certificate, so it passed it’s validity checks)
- Stopped and Started ALL Lync Services via PowerShell ( Stop/Start-CsWindowsService)
- Rebooted the Server (I know, I know, fixes everything but resolves nothing – but it didn’t fix it in this case).
So then I went down the route of looking at the permissions part of Event 14694 above.. “The certificate may have been deleted, invalid, or permissions are not set correctly“.
On the certificate in question, I right-clicked, went to All Tasks, and then Manage Private Keys, which showed that SYSTEM, Network Service, RTC Local Read-only Administrators, RTC Local User Administrators, and RTC Server Local Group all had the correct permissions.. And just to be thorough, I also checked manually after figuring out which key was the correct one (and where Windows keeps them)…
|
1 2 3 4 |
$certificate = (Get-ChildItem -Path CERT: | Where-Object {$_.SerialNumber -eq "7841FC7ABF2025"}) $keyfile = $certificate.PrivateKey.CspKeyContainerInfo.UniqueKeyContainerName (Get-Acl "C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\$keyfile").Access | FT -AutoSize |
To recap. No permission issues, the certificate was valid, and the chain was trusted all the way to the top.
My final leap of faith led me to services.msc where I finally resolved the issue. It’s a shame after all the troubleshooting, that the fix is so simple. I went to the Log On tab of Lync Server Access Edge service, cleared out both password fields and re-applied the Network Service credentials.
I’m sorry to say, all clues up until now were pretty much useless. Nothing to do with the certificate at all. But I learnt a few nice PowerShell tricks with CERT: and Get-Acl along the way.
I had this error crop up in the middle of a day with no explanation. Everything checked out, and none of the solutions appeared to help. I ended up explicitly adding an intermediate certificate to the trusted roots store (its issuing cert was there, which should be sufficient, but not it itself), which apparently resolved it — weird. Even stranger, nothing happened to its identically-configured peer.
As for the reversed certificate serials, unless I’m mistaken it wasn’t literally reversed, but reversed by octet — that is, 7841FC7ABF2025 should have been represented in the event log as 2520BF7AFC4178. (This kind of reversal is what I see in my environment.) I think this would likely be explained by byte ordering — big-endian vs. little-endian: http://en.wikipedia.org/wiki/Endianness
Thanks Jima, adding the intermediate cert to the trusted roots store got me going quickly again too.
Thx, adding intermediate to root store worked for us too.
Hi – Ditto on the octets. Also that the permissions / Network Service login weren’t the culprits for me. Instead…I simply had to add my GoDaddy *intermediate* cert to my “Intermediate Certification Authorities” truststore. Then the service started fine.
But still no clue on my side why it would – all of a sudden – want to do this to me. Thanks, M$ !!
Thx for this hint, this really fixed the problem here too…. Strange things happen…
Adding the intermediate certificate as trusted root resolved the problem here.
Spot on Jima – another case where it decided out of the blue that the certificate doesn’t work, and importing the Go Daddy intermediate certificate did the trick! Thanks very much.