MS14-055 Broke My Edge Server

** Update 5 see’s Microsoft re-release the server.msp security patch – confirmed working.

A client just contacted me saying they had no edge access. Internal clients were showing “Limited External Calling”.

Environment is Lync Server 2010, running on Windows 2008 R2.. This specifically affected the Edge Server role only.

Troubleshooting

Upon further inspection on the Edge Server, the services had stopped and were refusing to start. One of the errors seemed vagule certificate related.

I have seen similar ‘service not starting’ issues before, usually after an update, or CU where simply running Set-CsCertificate with the same thumbprint allows you to start the service again.. And it did, for the Access Edge, and suddenly federated presence worked again. But the AV Edge and AV Auth services still refused to start.

Because the Access Edge service was now running, the Lync Event Logs were showing Event ID 50007 LS AppDomain Host Process , saying that RtcHost.exe had stopped and would automatically be restarted.

And…

Also Windows Error Reporting events were flooding in too…

Windows (didn’t) Update.

After checking the Windows Update History there were a handful of windows update that had been installed automatically (client company’s practice for servers in DMZ), which included 2x Lync Server Updates

  • KB2953590 for OCSCore
  • KB2982385 for Server

Downloading the LyncServerUpdateInstaller.exe from MS was able to install OCSCore successfully, but we weren’t so lucky for KB2982385 – which showed the following dialog box.

Lync-Server-Update---Security-Warning

The client is currently talking to Microsoft to get an answer on this. I will update this article when I get more information.

Update 1

This issue has been confirmed by a few people who commented on this article, and replied on Twitter, as well as replicated in a lab by my colleague

The certificate of the patch has been confirmed trusted, and valid on the servers trying to install them.

Lync-Security-Patch-Certificate---server-mspUpdate 2

The UAC popup references G:\PreRelease\

Lync-Security-Patch-pre-release

Update 3

Microsoft Support recommended that my client should just ignore and install the patch, however I’m not sure that is the best thing to do, getting into a routine of dismissing security warnings (even if false-positive) is a bad idea, there must be a reason it failed. Microsoft Support then recommended disabling Certificate Revocation checking in Internet Explorer.

However, the certificate contained the following CRL locations:

  • http://crl.microsoft.com/pki/crl/products/MicCodSigPCA_08-31-2010.crl
  • http://crl.microsoft.com/pki/crl/products/MicrosoftTimeStampPCA.crl
  • http://crl.microsoft.com/pki/crl/products/microsoftrootcert.crl

And they were all accessible from the server, and didn’t contain a revocation for the certificate in question, nothing failed there.

I guess the package could contain something else with another digital signature, but I still maintain a Microsoft written patch for a Microsoft product, on a Microsoft platform, should not fail to verify. Something else must be going on here.

Update 4

Microsoft have removed the download to the server.msp patch file due to this ‘Known Issue’. More information will be added when available. The FAQ section of the security bulliten has been updated to include…

Why was this bulletin revised on September 15, 2014?
Microsoft revised this bulletin to address a known issue that prevented users from successfully installing security update 2982385 for Microsoft Lync Server 2010. Microsoft is investigating behavior associated with the installation of this update, and will update this bulletin when more information becomes available. As an added precaution, Microsoft has removed the download links to the 2982385 security update.

And the “Security Update for Lync 2010 (KB2982385, KB2982388)” now only includes rgs.msp

Update 5

And it’s back! Microsoft have yet again updated their security bulliten for MS14-055 with the following message

Why was this bulletin revised on September 23, 2014?
Microsoft rereleased this bulletin to announce the reoffering of the 2982385 security update file (server.msp) for Microsoft Lync Server 2010. The rereleased update addresses an issue in the original offering that prevented users from successfully installing the server.msp file. Customers who attempted to install the original update will be reoffered the 2982385 update and are encouraged to apply it at the earliest opportunity.

And there has already been one confirmed case of this being installed successfully. So it looks like we’re back on track.

Feel free to let me know what happens after you’ve grabbed the latest version.

Links

Vulnerabilities in Microsoft Lync Server Could Allow Denial of Service (2990928)

Thanks

With special thanks to @tobiefysh for supplying some information, screenshots, and yet another topic for my blog 🙂

Tweet about this on TwitterShare on LinkedInShare on Facebook
Pin on PinterestShare on Google+Digg thisShare on RedditShare on StumbleUponEmail this to someone

About Graham Cropley

Working as a Senior Consultant for Skype for Business, Exchange, and Office 365.

16 Comments

  1. Hi,
    I had the exact same issue, installing the KB2982385 and approving the Windows Security warning solves the issues.
    After approving the warning and installing the driver I was able to start the services.
    Microsoft should fix this warning or at least instruct users to what to do with the warning.

  2. I had exactly this issue with a customer after this Tuesdays auto updates. Looks to be .net related, fixed Uninstaling updates (and disabling auto updates) uninstalling Lync from the edge servers (2 in a pool) removing and reading .net then reinstalling Lync, importing configuration to local cms and re running deployment. Worked a treat on both. Probably overkill but getting it up and running was the priority rather than identifying the exact update and cause.

  3. Nothing helps! I can’t manually install the Update, it causes an Error. My Edge is down!!

    Why is the Driver Message Warning never shown on my Server? I got this Error Message with LyncServerUpdateInstaller.exe: – http://www11.pic-upload.de/15.09.14/ocwd4xedldan.png

  4. [15.09.2014 16:45:38] Starting Microsoft Lync Server 2010 Cumulative Update Installer, version 4.0.7577.276
    [15.09.2014 16:45:38] Embedded patch OcsCore.msp is at version 4.0.7577.276
    [15.09.2014 16:45:38] Product with GUID {A766C25B-A1D1-4711-A726-AC3E7CA4AAB3} is at version 4.0.7577.225 and is associated with patch OcsCore.msp which this installer has at version 4.0.7577.276
    [15.09.2014 16:45:38] OcsCore.msp, version 4.0.7577.276 is applicable.
    [15.09.2014 16:45:38] OcsCore.msp, version 4.0.7577.276 is NOT up-to-date on this server.
    [15.09.2014 16:45:39] Embedded patch Server.msp is at version 4.0.7577.276
    [15.09.2014 16:45:39] Product with GUID {A593FD00-64F1-4288-A6F4-E699ED9DCA35} is at version 4.0.7577.230 and is associated with patch Server.msp which this installer has at version 4.0.7577.276
    [15.09.2014 16:45:39] Server.msp, version 4.0.7577.276 is applicable.
    [15.09.2014 16:45:39] Server.msp, version 4.0.7577.276 is NOT up-to-date on this server.
    [15.09.2014 16:45:39] Embedded patch OcsCore.msp is referred by KB #2953590
    [15.09.2014 16:45:39] Embedded patch OcsCore.msp can be researched at URL http://support.microsoft.com/?kbid=2953590
    [15.09.2014 16:45:39] Embedded patch description: Update for Core Components
    [15.09.2014 16:45:39] Embedded patch Server.msp is referred by KB #2982385
    [15.09.2014 16:45:39] Embedded patch Server.msp can be researched at URL http://support.microsoft.com/?kbid=2982385
    [15.09.2014 16:45:39] Embedded patch description: Update for Lync Server 2010
    [15.09.2014 16:45:42] Beginning installation of selected binaries…
    [15.09.2014 16:45:42] Executing command: msiexec.exe /update “C:\Users\Administrator\Downloads\OcsCore.msp” /passive /norestart /l*vx “C:\Users\Administrator\Downloads\OcsCore.msp-XXXXXXXX-[2014-09-15][16-45-42]_log.txt”
    [15.09.2014 16:46:00] DONE: Installing KB2953590 for OcsCore.msp was installed successfully.
    [15.09.2014 16:46:00] Executing command: msiexec.exe /update “C:\Users\Administrator\Downloads\Server.msp” /passive /norestart /l*vx “C:\Users\Administrator\Downloads\Server.msp-XXXXXXXX-[2014-09-15][16-46-00]_log.txt”
    [15.09.2014 16:48:27] ERROR 1603: Server.msp had errors installing.
    [15.09.2014 17:02:31] ERROR: Lync ServerUpdateInstaller failed to successfully install all patches

    • And what about the contents of “C:\Users\Administrator\Downloads\Server.msp-XXXXXXXX-[2014-09-15][16-46-00]_log.txt”. Not that I’m expecting anything useful, but you never know.

  5. I have solved the problem 🙂 Give me a few minutes. I will explain.

  6. I had to deactivate the Sig Checking at Bootup http://www.pic-upload.de/view-24617511/DriverSigBoot.png.html
    I’ve tried this for fun, the Driver Screen in Windows apears, and now everything works 🙂 Oh my Godness.

  7. very helpful doc. This resolved my issue.

  8. Yes, after the Installation, I have rebooted the System with enabled Sig Checking. I know it’s not the best way, but tell me whats the alternative?? (Wait for a new MS Fix? MS doesn’t tell anything about this problem) I am really happy that my System is now working fine.

    • I guess that’s the only way to do it in Windows Server 2012. Thanks for posting your solution it may help others, glad you found a fix for it.. Hopefully Microsoft will re-release the patch soon.

  9. I’m having the same problems. Everything was working fine until late last week and now my Lync Server Audio/Video Authentication and Lync Server Audio/Video Edge services won’t start. But when I try to run the RGS, it tells me “The upgrade patch cannot be installed by the Windows Installer Service because the program to be upgraded may be missing, or the upgrade patch may update a different version of the program. Verify that the program to be upgraded exists on your computer and that you have the correct upgrade patch.” When I run the Cumulative Update Installer, it shows my update and installed version of Lync Server 2010 is 4.0.7577.230. I’m using Lync 2010 and Server 2008 so I don’t know what the problem is. Any idea what I’ve done wrong here?

    • The RGS patch is specifically for the Front End server which runs the Response Group Service. If you run LyncServerUpdateInstaller.exe it should only show you the relevant patches for the server it’s running on.. (i.e. on the Edge in this case I think it’s just OcsCore, UCMARuntime, and Server… and it’s Server that’s failing in this case). But Microsoft have removed the download for server.msp, so I would suggest try removing the other patches to rollback, and that should let you start the services. Let us know how you get on.

  10. Thanks Peter. Solved my problem.

Leave a Reply

Your email address will not be published. Required fields are marked *