Persistent Chat – December 2014 CU – 500 Internal Server Error

This issue was brought to my attention by Mika Ullgren (known as @Techmikal on Twitter – an expert in Security and Authentication). The issue was on a Lync Standard Edition Server with collocated Persistent Chat. After applying the latest December 2014 v2 Cumulative Update, the Persistent Chat webpage didn’t work externally, it just returned “500 – Internal Server Error”. But internally everything was still fine.

I checked it against my lab running the older November 2014 CU, and mine worked externally. So something has changed between the two CUs that has broken this functionality.

Comparing The Differences

The first obvious thing was that the Internal and External configuration in IIS differed.. Internally, Persistent Chat is configured as an Application, externally it is a Virtual Directory. But this is the same at both patch levels, even the RTM version of Lync Web Components creates the external Persistent Chat as a Virtual Directory. So why is that a problem now? Well that’s what I wanted to find out.

Enabling IIS Detailed Error Pages

“500” is not a lot to go on. By default IIS will not show detailed errors to remote users, let’s turn that on, and see what we get…

Open up Internet Information Services (IIS) Manager, and either on the Site (Lync Server External Web Site), or the root of IIS, select Error Pages, and then click on Edit Feature Settings… under the Actions pane on the right.

Select Detailed Errors and click ok.

Now when you refresh the browser you get a bit more information.

We can now see that it’s actually 500.19 with HRESULT code of 0x80070021, (you can click the View More Information » link at the bottom of the page and lookup the error for some helpful, but generic information).

The important bits here were…

and this…

So What’s Changed?

Well not much on the face of it… it looks like the culprit is in the web.config file from the later CU in %ProgramFiles%\Microsoft Lync Server 2013\Web Components\PersistentChat\Ext , which includes these extra lines inside the  <configuration>  element…

This would seem to ensure that anything called PreAuthModule that may be there already is removed, and the Microsoft.Rtc.Internal.PreAuthModule was registered instead. Also by removing it first, this won’t cause an error from that module already being added.

Fix 1

Converting the PersistentChat Virtual Directory under the External Web Site to an Application resolves the issue. And I’m able to log in. Surely that can’t be the best fix? Although, all the other items under there are Applications, why not Persistent Chat?

Fix 2

Ok, instead of that.. Try removing the entire  <system.webServer> section, so that it doesn’t try to remove/add any modules. Without those lines, I’m still able to log in (using Forms Based Authentication, just like before). Essentially reverting to the November 2014 CU version of the  web.config file. But that can’t the best fix either? Removing an authentication module? That can’t help surely? But it doesn’t seem to break anything with the limited testing I performed.

Fix 3

If you’ve read my blog before, you’ll know I just couldn’t leave it there. I didn’t really want to remove bits of fine hand crafted code that Microsoft saw it fit to add. After all, removing reference to an Authentication Module doesn’t feel right. On the other hand, changing a Virtual Directory, which has always been a Virtual Directory right from RTM doesn’t see quite right either.

Remember when I picked out the important bits from the detailed error page? Well, in IIS there is a concept of ‘locking’ configuration, at different levels, for different things. And the error above basically says. “The configuration is invalid, because you are not allowed to make that change here, at this level”. If you’re familiar with httpd.conf and .htaccess files in Apache, you’ll probably understand this concept already.

So finally, another way to get this working is to tell IIS, that configuration changes to  <modules>  under  <system.webServer>are allowed below an Application, which is just what we’ve got here. The PersistentChat object is a Virtual Directory underneath the root Application. Not an application itself.

If you open  %windir%\System32\inetsrv\config\applicationHost.config look for  <configSections> <sectionGroup name="system.webServer">  and then the line that starts <section name="modules". Here’s an example…

The very last line in the above code is what we’re looking at. It’s currently set with allowDefinition as "MachineToApplication", at the top of this file is a short list, but a better description is available on MSDN, the options are…

Changing that line to the following, also lets Persistent Chat Web site to work, and I can log in again.

But, that allows ANY Virtual Directory under ANY Application to set modules.

Conclusion

Until Microsoft release an updated patch. At the moment my best guess is to go with Fix 1 – converting the Virtual Directory to an Application. Because it has the least impact, and smallest scope for change.

Does anybody have any background information, why Persistent Chat is a Virtual Directory under the External Site? Or perhaps some detail on what this new PreAuthModule does?

Thanks