Persistent Chat – December 2014 CU – 500 Internal Server Error

This issue was brought to my attention by Mika Ullgren (known as @Techmikal on Twitter – an expert in Security and Authentication). The issue was on a Lync Standard Edition Server with collocated Persistent Chat. After applying the latest December 2014 v2 Cumulative Update, the Persistent Chat webpage didn’t work externally, it just returned “500 – Internal Server Error”. But internally everything was still fine.

IIS-500-Internal-Server-ErrorI checked it against my lab running the older November 2014 CU, and mine worked externally. So something has changed between the two CUs that has broken this functionality.

Comparing The Differences

The first obvious thing was that the Internal and External configuration in IIS differed.. Internally, Persistent Chat is configured as an Application, externally it is a Virtual Directory. But this is the same at both patch levels, even the RTM version of Lync Web Components creates the external Persistent Chat as a Virtual Directory. So why is that a problem now? Well that’s what I wanted to find out.

Enabling IIS Detailed Error Pages

“500” is not a lot to go on. By default IIS will not show detailed errors to remote users, let’s turn that on, and see what we get…

Open up Internet Information Services (IIS) Manager, and either on the Site (Lync Server External Web Site), or the root of IIS, select Error Pages, and then click on Edit Feature Settings… under the Actions pane on the right.

Select Detailed Errors and click ok.

IIS-Detailed-Error-Responses

Now when you refresh the browser you get a bit more information.

IIS-500-19-Detailed-ErrorWe can now see that it’s actually 500.19 with HRESULT code of 0x80070021, (you can click the View More Information » link at the bottom of the page and lookup the error for some helpful, but generic information).

The important bits here were…

and this…

So What’s Changed?

Well not much on the face of it… it looks like the culprit is in the web.config file from the later CU in %ProgramFiles%\Microsoft Lync Server 2013\Web Components\PersistentChat\Ext , which includes these extra lines inside the  <configuration>  element…

This would seem to ensure that anything called PreAuthModule that may be there already is removed, and the Microsoft.Rtc.Internal.PreAuthModule was registered instead. Also by removing it first, this won’t cause an error from that module already being added.

Fix 1

Converting the PersistentChat Virtual Directory under the External Web Site to an Application resolves the issue. And I’m able to log in. Surely that can’t be the best fix? Although, all the other items under there are Applications, why not Persistent Chat?

IIS-Convert-To-Application

Fix 2

Ok, instead of that.. Try removing the entire  <system.webServer> section, so that it doesn’t try to remove/add any modules. Without those lines, I’m still able to log in (using Forms Based Authentication, just like before). Essentially reverting to the November 2014 CU version of the  web.config file. But that can’t the best fix either? Removing an authentication module? That can’t help surely? But it doesn’t seem to break anything with the limited testing I performed.

Fix 3

If you’ve read my blog before, you’ll know I just couldn’t leave it there. I didn’t really want to remove bits of fine hand crafted code that Microsoft saw it fit to add. After all, removing reference to an Authentication Module doesn’t feel right. On the other hand, changing a Virtual Directory, which has always been a Virtual Directory right from RTM doesn’t see quite right either.

Remember when I picked out the important bits from the detailed error page? Well, in IIS there is a concept of ‘locking’ configuration, at different levels, for different things. And the error above basically says. “The configuration is invalid, because you are not allowed to make that change here, at this level”. If you’re familiar with httpd.conf and .htaccess files in Apache, you’ll probably understand this concept already.

So finally, another way to get this working is to tell IIS, that configuration changes to  <modules>  under  <system.webServer>are allowed below an Application, which is just what we’ve got here. The PersistentChat object is a Virtual Directory underneath the root Application. Not an application itself.

If you open  %windir%\System32\inetsrv\config\applicationHost.config look for  <configSections> <sectionGroup name="system.webServer">  and then the line that starts <section name="modules". Here’s an example…

The very last line in the above code is what we’re looking at. It’s currently set with allowDefinition as "MachineToApplication", at the top of this file is a short list, but a better description is available on MSDN, the options are…

Value Definition
“Everywhere” The section can be defined in any configuration level. The default value.
“MachineToApplication” The section can be defined in the Machine.config or ApplicationHost.config file.
“MachineOnly” The section can be defined only in the Machine.config file.
“MachineToWebRoot” The section can be defined in the Machine.config, ApplicationHost.config, or Web.config file. The Web.config file is stored in the Web site root.
“AppHostOnly” The section can be defined only in the ApplicationHost.config file.

Changing that line to the following, also lets Persistent Chat Web site to work, and I can log in again.

But, that allows ANY Virtual Directory under ANY Application to set modules.

Conclusion

Until Microsoft release an updated patch. At the moment my best guess is to go with Fix 1 – converting the Virtual Directory to an Application. Because it has the least impact, and smallest scope for change.

Does anybody have any background information, why Persistent Chat is a Virtual Directory under the External Site? Or perhaps some detail on what this new PreAuthModule does?

Thanks

Tweet about this on TwitterShare on LinkedInShare on Facebook
Pin on PinterestShare on Google+Digg thisShare on RedditShare on StumbleUponEmail this to someone

About Graham Cropley

Working as a Senior Consultant for Skype for Business, Exchange, and Office 365.

5 Comments

  1. Greg S (@GregSeeber)

    That is pretty awesome of you to catch that. Wow. I googled “500 internal server error” persistent chat” and got your blog. I was doing a demo on PCHAT and was really embarrased when I had to connect to VPN and hit the internal web services to get this working. *zzzzz*. Thanks so much.

  2. Excellent info. We just saw this at a client’s installation. This was not collocated. Dedicated PChat machines. Just for clarification to those in a similar situation, the VDirs refereced here are on the FE, not the PChat servers.

  3. Courtney Culver

    Thanks for the post on this – it definitely helped me out! One thing I wanted to add, if you have multiple FEs (Like I do) then you must convert each Virtual Directory to an Application on each Front End. Once you’ve made the change on each FE, you should get the Persistent Chat site back up and running.

  4. Thanks for this post. P-Chat site back and working fine.

  5. Another excellent post Graham. I didn’t know I had this issue on my lab servers. Fix 1 was my choice and it worked perfectly of course. Thanks again.

Leave a Reply

Your email address will not be published. Required fields are marked *