Lync Edge – January 2014 CU Update Issue

This one will be short I promise. After installing Lync Server 2013 CU4 (January 2014), everything worked fine on the Front Ends, but ran into a rather weird issue with an Edge Server.

After installing the update, the Lync Server Access Edge service refused to start, all other services started ok. It bombed out pretty quickly with the following error…

And Event Viewer recorded these…

Event 12303, LS Server

Event 14694, LS Protocol Stack

Event 14397, LS Protocol Stack

How bizarre, it had been working fine for several months, nothing was due to expire.. Quickly checking the Local Machine’s certificate store reassured me the certificate and it’s full chain were valid and trusted.

I’d just like to add, the Serial Number supplied in the Details tab of the event log didn’t match anything on the server.. (this turned into a little detour). The Serial Number in the Event Log was 5202FBA7CF1487, and the Serial Number assigned to the external certificate was 7841FC7ABF2025 (Names changed to protect the innocent)… Running the following PowerShell confirmed that the Serial Number simply doesn’t exist.

Wait… Any body spot it? Is it glaringly obvious? It certainly wasn’t at first… but then it hit me… It’s backwards. The serial number displayed in Event Viewer is backwards… Why would anybody be so inclined to display a descriptive error message containing a certificate Serial Number, for a certificate it was trying to identify, in reverse?!? Anyway.. so it DOES exist, and as I saw earlier it was valid and trusted, and now I know we’re talking about the same certificate.

I can’t honestly remember each thing I tried before I stumbled on the fix., here are some of the attempts that failed to do anything:

  • Ensured topology and CMS were replicated, and all UpToDate. (It was, and everything was replicating fine).
  • Removed, Deleted, re-imported, and re-assigned the certificate. (Deployment Wizard showed me the certificate, so it passed it’s validity checks)
  • Stopped and Started ALL Lync Services via PowerShell ( Stop/Start-CsWindowsService)
  • Rebooted the Server (I know, I know, fixes everything but resolves nothing – but it didn’t fix it in this case).

So then I went down the route of looking at the permissions part of Event 14694 above.. “The certificate may have been deleted, invalid, or permissions are not set correctly“.

On the certificate in question, I right-clicked, went to All Tasks, and then Manage Private Keys, which showed that SYSTEM, Network Service, RTC Local Read-only Administrators, RTC Local User Administrators, and RTC Server Local Group all had the correct permissions.. And just to be thorough, I also checked manually after figuring out which key was the correct one (and where Windows keeps them)…

To recap. No permission issues, the certificate was valid, and the chain was trusted all the way to the top.

My final leap of faith led me to services.msc where I finally resolved the issue. It’s a shame after all the troubleshooting, that the fix is so simple. I went to the Log On tab of Lync Server Access Edge service, cleared out both password fields and re-applied the Network Service credentials.

Lync-Access-Edge-Running

I’m sorry to say, all clues up until now were pretty much useless. Nothing to do with the certificate at all. But I learnt a few nice PowerShell tricks with  CERT: and  Get-Acl along the way.

Tweet about this on TwitterShare on LinkedInShare on Facebook
Pin on PinterestShare on Google+Digg thisShare on RedditShare on StumbleUponEmail this to someone

About Graham Cropley

Working as a Senior Consultant for Skype for Business, Exchange, and Office 365.

7 Comments

  1. I had this error crop up in the middle of a day with no explanation. Everything checked out, and none of the solutions appeared to help. I ended up explicitly adding an intermediate certificate to the trusted roots store (its issuing cert was there, which should be sufficient, but not it itself), which apparently resolved it — weird. Even stranger, nothing happened to its identically-configured peer.

    As for the reversed certificate serials, unless I’m mistaken it wasn’t literally reversed, but reversed by octet — that is, 7841FC7ABF2025 should have been represented in the event log as 2520BF7AFC4178. (This kind of reversal is what I see in my environment.) I think this would likely be explained by byte ordering — big-endian vs. little-endian: http://en.wikipedia.org/wiki/Endianness

  2. Hi – Ditto on the octets. Also that the permissions / Network Service login weren’t the culprits for me. Instead…I simply had to add my GoDaddy *intermediate* cert to my “Intermediate Certification Authorities” truststore. Then the service started fine.

    But still no clue on my side why it would – all of a sudden – want to do this to me. Thanks, M$ !!

  3. Adding the intermediate certificate as trusted root resolved the problem here.

  4. Spot on Jima – another case where it decided out of the blue that the certificate doesn’t work, and importing the Go Daddy intermediate certificate did the trick! Thanks very much.

Leave a Reply

Your email address will not be published. Required fields are marked *