This issue was brought to my attention by Mika Ullgren (known as @Techmikal on Twitter – an expert in Security and Authentication). The issue was on a Lync Standard Edition Server with collocated Persistent Chat. After applying the latest December 2014 v2 Cumulative Update, the Persistent Chat webpage didn’t work externally, it just returned “500 – Internal Server Error”. But internally everything was still fine.
I checked it against my lab running the older November 2014 CU, and mine worked externally. So something has changed between the two CUs that has broken this functionality.
Comparing The Differences
The first obvious thing was that the Internal and External configuration in IIS differed.. Internally, Persistent Chat is configured as an Application, externally it is a Virtual Directory. But this is the same at both patch levels, even the RTM version of Lync Web Components creates the external Persistent Chat as a Virtual Directory. So why is that a problem now? Well that’s what I wanted to find out.
Enabling IIS Detailed Error Pages
“500” is not a lot to go on. By default IIS will not show detailed errors to remote users, let’s turn that on, and see what we get…
Open up Internet Information Services (IIS) Manager, and either on the Site (Lync Server External Web Site), or the root of IIS, select Error Pages, and then click on Edit Feature Settings… under the Actions pane on the right.
Select Detailed Errors and click ok.
Now when you refresh the browser you get a bit more information.
We can now see that it’s actually 500.19 with HRESULT code of 0x80070021, (you can click the View More Information » link at the bottom of the page and lookup the error for some helpful, but generic information).
The important bits here were…
1 |
The requested page cannot be accessed because the related configuration data for the page is invalid. |
and this…
1 |
Config Error: Configuration section not allowed to be set below application |
So What’s Changed?
Well not much on the face of it… it looks like the culprit is in the web.config file from the later CU in %ProgramFiles%\Microsoft Lync Server 2013\Web Components\PersistentChat\Ext , which includes these extra lines inside the <configuration> element…
1 2 3 4 5 6 |
<system.webServer> <modules> <remove name="PreAuthModule" /> <add name="PreAuthModule" type="Microsoft.Rtc.Internal.PreAuthModule.PreAuthModule,Microsoft.Rtc.Server.WebInfrastructure, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/> </modules> </system.webServer> |
This would seem to ensure that anything called PreAuthModule that may be there already is removed, and the Microsoft.Rtc.Internal.PreAuthModule was registered instead. Also by removing it first, this won’t cause an error from that module already being added.
Fix 1
Converting the PersistentChat Virtual Directory under the External Web Site to an Application resolves the issue. And I’m able to log in. Surely that can’t be the best fix? Although, all the other items under there are Applications, why not Persistent Chat?
Fix 2
Ok, instead of that.. Try removing the entire <system.webServer> section, so that it doesn’t try to remove/add any modules. Without those lines, I’m still able to log in (using Forms Based Authentication, just like before). Essentially reverting to the November 2014 CU version of the web.config file. But that can’t the best fix either? Removing an authentication module? That can’t help surely? But it doesn’t seem to break anything with the limited testing I performed.
Fix 3
If you’ve read my blog before, you’ll know I just couldn’t leave it there. I didn’t really want to remove bits of fine hand crafted code that Microsoft saw it fit to add. After all, removing reference to an Authentication Module doesn’t feel right. On the other hand, changing a Virtual Directory, which has always been a Virtual Directory right from RTM doesn’t see quite right either.
Remember when I picked out the important bits from the detailed error page? Well, in IIS there is a concept of ‘locking’ configuration, at different levels, for different things. And the error above basically says. “The configuration is invalid, because you are not allowed to make that change here, at this level”. If you’re familiar with httpd.conf and .htaccess files in Apache, you’ll probably understand this concept already.
So finally, another way to get this working is to tell IIS, that configuration changes to <modules> under <system.webServer>are allowed below an Application, which is just what we’ve got here. The PersistentChat object is a Virtual Directory underneath the root Application. Not an application itself.
If you open %windir%\System32\inetsrv\config\applicationHost.config look for <configSections> <sectionGroup name="system.webServer"> and then the line that starts <section name="modules". Here’s an example…
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
<configSections> <sectionGroup name="system.applicationHost"> <section name="applicationPools" allowDefinition="AppHostOnly" overrideModeDefault="Deny" /> <section name="configHistory" allowDefinition="AppHostOnly" overrideModeDefault="Deny" /> <section name="customMetadata" allowDefinition="AppHostOnly" overrideModeDefault="Deny" /> <section name="listenerAdapters" allowDefinition="AppHostOnly" overrideModeDefault="Deny" /> <section name="log" allowDefinition="AppHostOnly" overrideModeDefault="Deny" /> <section name="serviceAutoStartProviders" allowDefinition="AppHostOnly" overrideModeDefault="Deny" /> <section name="sites" allowDefinition="AppHostOnly" overrideModeDefault="Deny" /> <section name="webLimits" allowDefinition="AppHostOnly" overrideModeDefault="Deny" /> </sectionGroup> <sectionGroup name="system.webServer"> <section name="asp" overrideModeDefault="Deny" /> <section name="caching" overrideModeDefault="Allow" /> <section name="cgi" overrideModeDefault="Deny" /> <section name="defaultDocument" overrideModeDefault="Allow" /> <section name="directoryBrowse" overrideModeDefault="Allow" /> <section name="fastCgi" allowDefinition="AppHostOnly" overrideModeDefault="Deny" /> <section name="globalModules" allowDefinition="AppHostOnly" overrideModeDefault="Deny" /> <section name="handlers" overrideModeDefault="Deny" /> <section name="httpCompression" allowDefinition="AppHostOnly" overrideModeDefault="Deny" /> <section name="httpErrors" overrideModeDefault="Allow" /> <section name="httpLogging" overrideModeDefault="Deny" /> <section name="httpProtocol" overrideModeDefault="Allow" /> <section name="httpRedirect" overrideModeDefault="Allow" /> <section name="httpTracing" overrideModeDefault="Deny" /> <section name="isapiFilters" allowDefinition="MachineToApplication" overrideModeDefault="Deny" /> <section name="modules" allowDefinition="MachineToApplication" overrideModeDefault="Deny" /> |
The very last line in the above code is what we’re looking at. It’s currently set with allowDefinition as "MachineToApplication", at the top of this file is a short list, but a better description is available on MSDN, the options are…
Value | Definition |
---|---|
“Everywhere” | The section can be defined in any configuration level. The default value. |
“MachineToApplication” | The section can be defined in the Machine.config or ApplicationHost.config file. |
“MachineOnly” | The section can be defined only in the Machine.config file. |
“MachineToWebRoot” | The section can be defined in the Machine.config, ApplicationHost.config, or Web.config file. The Web.config file is stored in the Web site root. |
“AppHostOnly” | The section can be defined only in the ApplicationHost.config file. |
Changing that line to the following, also lets Persistent Chat Web site to work, and I can log in again.
1 |
<section name="modules" allowDefinition="Everywhere" overrideModeDefault="Deny" /> |
But, that allows ANY Virtual Directory under ANY Application to set modules.
Conclusion
Until Microsoft release an updated patch. At the moment my best guess is to go with Fix 1 – converting the Virtual Directory to an Application. Because it has the least impact, and smallest scope for change.
Does anybody have any background information, why Persistent Chat is a Virtual Directory under the External Site? Or perhaps some detail on what this new PreAuthModule does?
That is pretty awesome of you to catch that. Wow. I googled “500 internal server error” persistent chat” and got your blog. I was doing a demo on PCHAT and was really embarrased when I had to connect to VPN and hit the internal web services to get this working. *zzzzz*. Thanks so much.
Excellent info. We just saw this at a client’s installation. This was not collocated. Dedicated PChat machines. Just for clarification to those in a similar situation, the VDirs refereced here are on the FE, not the PChat servers.
Thanks for the post on this – it definitely helped me out! One thing I wanted to add, if you have multiple FEs (Like I do) then you must convert each Virtual Directory to an Application on each Front End. Once you’ve made the change on each FE, you should get the Persistent Chat site back up and running.
Thanks for this post. P-Chat site back and working fine.
Another excellent post Graham. I didn’t know I had this issue on my lab servers. Fix 1 was my choice and it worked perfectly of course. Thanks again.