LyncDiscover and Auto Discovery Deeper Dive

I’m sure everybody has used Microsoft’s Remote Connectivity Analyzer for Exchange / ActiveSync as well as Lync / OCS. It works by stepping through the discovery and connection process. Very useful, but not very controllable, and it’s only for external use.

Some of you may have also come across the Lync Connectivity Analyzer, it’s an Offline tool that runs on your PC, and is primarily designed to troubleshoot connection issues with Windows Store App and others that use the LyncDiscover service. Sounds useful, but I’ve never been able to get it to actually work – there is a seemingly well-known bug between it and .NET 4.5.1, so if you’re running Windows 8.1 (like I am), there’s no hope until a new version of the tool is released as you can’t roll back to an earlier version of .NET. There appears to be hope if you are running Windows 7 or Windows 8.0, which is to remove a hot-fix (for more information see here, and here) to get the tool to actually run (with limited success it seems). I’ve regularly checked to see if Lync Connectivity Analyser has been fixed, and for the past 6 months it’s crashed every time.

** UPDATE ** – NextHop released an updated version of the Lync Connectivity Analyser shortly after I posted this. Which is available here. However it still crashes on my Windows 8.1 machine.

I finally cracked, and set myself on a mission to write my own application, (this will be released in another blog post – coming soon). Sneaky screenshot of “Lyncer – Lync Tester”.

Over the past couple of weeks I’ve come to realise there is very little in the way of detailed information on LyncDiscover / Auto Discovery process past a certain point. The best I’ve found is Lync Server 2010 – Mobility Deep Dive – Autodiscover Service on the NextHop TechNet Blog.

Everybody who’s reading this has at some point, manually typed https://lyncdiscover.yourdomain.co.uk/ into a browser and seen a JSON or XML formatted response and thought.. “good, that works”.

There are plenty of fantastic articles around for describing how LyncDiscover plays a part, and the general steps in big picture of the client discovery process. But that wasn’t enough for me…

With this article I intend to take it just that little bit further.

Recap

I won’t go into too much detail just yet, as the basics are well documented. The client discovery process involves looking for the following DNS Entries in order. (I won’t be covering SRV Records in this article).

  • lyncdiscoverinternal.sipdomain.co.uk
  • lyncdiscover.sipdomain.co.uk

And the client then tries HTTP (Port 80) first and then HTTPS (Port 443) for each FQDN and (optionally) appends the SIP URI in the HTTP GET request as follows

It’s important that either of the following two HTTP Headers are present, as the server is capable of outputting XML or JSON formatted responses.

Ask for JSON…

The reason for JSON (I surmise) is partly because it’s easier to parse than XML, but mainly due to Unified Communications Web API (UCWA). This is a new web services (REST API) built into Lync 2013, designed with standards such as HTTP, JSON, and JavaScript which expose Instant Messaging, Presence, and more. Microsoft have done a good job providing plenty of information and examples on ucwa.lync.com as well as interactive demo’s allowing your code to authenticate into a pre-populated sandbox (“gotuc.net”) – a nice touch I think. More on OAuth Tokens and Authentication later.

For JSON, Include the following HTTP Header in the request

This is for Lync Server 2013 only, Lync Server 2010 does not include UCWA (nor OAuth), so the server will response with HTTP 406 – Not Acceptable.

and the response looks like this:

Because the request was for a JSON formatted response, the LyncDiscover service assumes you’re going to be using UCWA and suggests the OAuth method for obtaining the user’s home pool. As you can see JSON actually returns less information than the XML response:

  • “self” is a reference to the web services that generated this response.
  • “xframe” is intended to be used by UCWA web-based applications and provides a way around the limitations of AJAX with cross-domain requests.
  • “user” is the particular URL we are interested in, it’s the auto discovery URL that uses OAuth Authentication.

Remember this ‘user’ URL and carry on reading…

Ask for XML…

This is the preferred format used, and provides more information

Lync Server 2010 response:

Lync Server 2013 response:

Lets talk it through, firstly the attribute AccessLocation  in the root AutodiscoverResponse  Element says “External” that’s useful information to know. As far as I can work out, this is simply picked from the server’s listening port number (443 for internal, 4443 for external). Later on this can be used that to identify the correct URL based on your location in relation to your home pool.

  • “Domain” is for discovering the FQDN of the current pool, and does not require authentication.
  • “User” is for discovering the home pool for the user, requires authentication.
  • “OAuth” is the same as “User” except it requires OAuth authentication instead of NTLM or Kerberos.

AutoDiscover with WebTicketService

In this little walk-through we are attempting to discover a specific users home pool, firstly the “User” method, so we send a HTTP GET to the “User” URL like this:

At which point we get HTTP Response of 401 Unauthorised letting us know that authentication is required, but it doesn’t accept any of the traditional methods of authentication, you cannot pass NTLM or Kerberos, as both of those require connection to AD to verify the user’s credentials, this would increase the reliance on AD, and would impose additional overhead for each connection attempt. Also NTLM over HTTP is quite bloated requiring each connection to authenticate fully.

The method of obtaining a Ticket allows you to prove your identity once, and receive a unique ticket that is valid for a set length of time, allowing you to simply pass your ticket in subsequent requests, making it much quicker and less resource intensive to make a connection without having to go through the entire re-authentication process each time (compared with NTLM).

Along with the 401 Unauthorised, the server returns with the following headers:

As well as seeing the internal FQDN of server that’s responding to this request, the important piece of information here is the  X-MS-WebTicketURL this gives you the location of the WebTicketService. This is a SOAP web service that authenticates a user via NTLM or Kerberos (if configured) and returns a SAML Assertion (Ticket) as part of the SOAP Message response.

This time we send a HTTP POST to the URL retrieved above, we again receive a 401 Unauthorised, but this time we see  WWW-Authenticate: NTLM, so lets do that, and after the challenge-response has completed, we also include the following header:

and attach our SOAP Message, invoking the “Issue” method of the WebTicketService.

This is a SOAP Message exchange that uses WS-Trust (an extension of the WS-Security specification) that provides a framework for exchanging security tokens and to establish a trust relationship.

The parts of the message that we’re interested in are the following 3 Elements:

  • “EndpointReference” contains the address of the web service we would like to access, using the ticket we receive next as our proof we’ve already authenticated.
  • “Claim” is the SIP Address of the user we are requesting the ticket for.
  • “Entropy” is passed as a “BinarySecret” because encryption is not needed as we’re talking over TLS. In this case a Nonce is passed which is just a Base64 encoded unique key which is used as ‘proof of possession’ to prevent replay attacks.

I’ve noticed that PreAuthentication is required, and that Chunking needs to be disabled (if the message is chunked, the server doesn’t response with 100 Continue as expected, but instead returns an error prematurely saying the request was invalid).

We then receive our SAML Assertion wrapped in a SOAP Message. Here’s the usual header / message descriptor:

The contents of the  RequestedSecurityToken element makes up our SAML Assertion, in essence, our Ticket.

Fantastic, we’ve now obtained something we can use to prove our identity. The way we do this is to literally Base64 encode the above SAML Assertion (everything between the  RequestedSecurityToken element), which becomes our Ticket.

Let’s go back to the “User” AutoDiscover URL

And this time we can provide authentication by adding our Ticket to the HTTP header as follows:

Everybody’s happy and we get our AutoDiscover XML, which we’ll go through after the next few section.

AutoDiscover with OAuth

OAuth is a much simpler way of obtaining a token which can be used to prove our identity. OAuth is an open standard written by Microsoft and published by the IETF as RFC6749. At a high level, it provides the same functionality as the WebTicketService above, but the most important difference to point out is the token received is created using the X509 Certificate assigned on the server, therefor allowing any server with that certificate to decode and trust the token without the need for the client to re-authenticate with another server. (Lync 2013 automatically replicates this OAuth certificate to all other Front End Servers via the CMS).

Start off by sending a GET to the “OAuth” URL obtained from the XML formatted LyncDiscover response (or the “User” URL from the JSON response):

We are once again sent back a 401 Unauthorised, this time with the following headers:

This time the  WWW-Authenticate header gives us a MsRtcOAuth URL of https://lyncweb.sipdomain.co.uk/WebTicket/oauthtoken which accepts several different “grant types”, we can see it accepts Windows Authentication, Password Authentication, and a grant type that allows users to join meetings anonymously.

All we need to do is build the following POST data

and send it to the “OAuth” URL.

The JSON response is as equally straightforward.

We simply use the value of “access_token” the same way we used the WebService Ticket above by sending a HTTP GET to the OAuth AutoDiscover URL again, as follows:

with the following header, which simply acts as our proof of identity.

So that’s another way to get the same AutoDiscover XML out of a Lync Server.

Not quite there yet….

What have we done so far, let’s remind ourselves.

  • Lookup the DNA A record for lyncdiscoverinternal.sipdomain.co.uk or lyncdiscover.sipdomain.co.uk
  • Get JSON or XML response which identifies the external web services of a Lync Server
  • Obtained a WebService Ticket, or an OAuth Token
  • And authenticated against the User or OAuth AutoDiscovery URL

If the user isn’t homed on the same Lync pool that the Reverse Proxy is forwarding these requests to, then we get back a redirect response. We need to follow that, and repeat the above steps to authenticate against the new server.

A JSON redirection looks like this

And an XML redirection looks like this.

AutoDiscover XML Response

Lync Server 2010 looks something like this:

Lync Server 2013 looks something like this

The first 4 lines under the “User” element tell you the internal FQDN of your home Front End Pool, and external Access Edge FQDN of the Edge Pool associated with your home pool and which port to use based on your type (client or server).

Then we get a rather self-explanatory list of Internal and corresponding External web service URLs for the following services:

  • “Autodiscover” – been there, done this, but we now know which server which we’re homed on, so we wouldn’t have to follow any redirects next time (unless we move).
  • “AuthBroker” is the URL for the Authentication Broker (Reach) web service.
  • “WebScheduler” is the URL for the Lync Web Scheduler service.
  • “CertProvisioning” is the Web Service for obtaining client certificates – (more on this in another article coming soon).
  • “MCX” is the Lync Mobility web services
  • “UCWA” is the new Unified Communication Web API intended to replace, and improve upon MCX.
  • “XFrame” used for cross-domain AJAX requests.
  • “Self” the Web Service that returned this page.

What have we learnt?

We now know two separate ways to obtain information about the home pool of a specific user and associated web services.

WebTicketService is available on Lync Server 2010 and 2013, and requires NTLM or Kerberos support to obtain the initial Ticket which can then be used to prove the identity of the user from then on.

OAuth is available only on Lync Server 2013, and provides an easier (and more web friendly) way to obtain a token, which can also be used to prove identity, on any Front End Server in the topology as they all share the same X509 Certificate, replicated via the CMS.

What’s next

Next up (as and when I get more time), I will talk about my experience working and testing the Access Edge using SIP (User registration, and some other commands such as OPTIONS, NOTIFY, and SUBSCRIBE). Including some detail on the protocols used to compress SIP Traffic which I personally found equally interesting as frustrating.

After that I plan to add service level tests to my program, going past just discovering, to actually test against MCX, CertProvisioning, and AuthBroker and to see that they are working as expected.

 

Thanks for reading, comments are always welcome, and encouraged 🙂

Tweet about this on TwitterShare on LinkedInShare on Facebook
Pin on PinterestShare on Google+Digg thisShare on RedditShare on StumbleUponEmail this to someone

About Graham Cropley

Working as a Senior Consultant for Skype for Business, Exchange, and Office 365.

10 Comments

  1. Hi if i browse to https://lyncdiscover.domain.com i get

    Server Error

    403 – Forbidden: Access is denied.
    You do not have permission to view this directory or page using the credentials that you supplied.

    • Look at the config on your reverse proxy to check things like pre-auth, authentication delegation, or pass-though is disabled, depending on what you are using. Also, maybe check IIS on your Front Ends and/or Directors and see if Anonymous Authentication is enabled for the Autodiscover virtual directory.

      Can you provide any more information?

  2. Brilliant article. Thank you for this. It’s been so helpful for understanding and troubleshooting issues.

  3. Hi Graham,

    Thanks for this informative article. I was running the request for AutoDiscover with WebTicketService. I generated the OAuth token using username password creds. My request keeps returning 401 unauthorized. I am unsure what am i missing here.

    Request:
    ======
    POST /WebTicket/WebTicketService.svc HTTP/1.1
    Host: webdir0m.online.lync.com
    Cache-Control: no-cache
    SoapAction: http://docs.oasis-open.org/ws-sx/ws-trust/200512/RST/Issue
    Content-Type: application/soap+xml; charset=utf-8
    Authorization: Bearer cwt=AAEBHAEFAAAAAAAFFtest123
    Postman-Token: cf1fadeb-35f5-5d1f-8c2a-cfaadd9bd3bb

    http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLV1.1
    http://docs.oasis-open.org/ws-sx/ws-trust/200512/RST/Issue

    https://webdir0m.online.lync.com/Autodiscover/AutodiscoverService.svc/root/user?sipuri=abc@microsoft.com

    sip:abc@microsoft.com

    OTJhYzRlZjMtYTY1OC00MGQ1LWI2YjQtNzczOGFlZDFmZjVl

    http://docs.oasis-open.org/ws-sx/ws-trust/200512/SymmetricKey

    Response Headers:
    =============
    Content-Length → 1293
    Content-Type → text/html
    Date → Wed, 22 Apr 2015 19:00:08 GMT
    Strict-Transport-Security → max-age=31536000; includeSubDomains
    WWW-Authenticate → NTLM
    X-Content-Type-Options → nosniff
    X-MS-Correlation-Id → 2147494807
    X-MS-Server-Fqdn → BN10M00EDG01.infra.lync.com
    client-request-id → 4bd04339-ec0e-4fe4-92db-94e053550d4b

    Will appreciate any response.

    Thanks,
    Tushar

    • my soap message got garbled in the above request. fyi.

    • It looks like you are trying this against Lync Online / Skype for Business Online.. It doesn’t support using that authentication method, as you can see from the reply, the only WWW-Authenticate option given to you is NTLM.

  4. How does lyncdiscover actually work? which port does it use? 80 or 443? I have a setup to redirect request on port 80 to 443 on another IP because i don’t want to update my certificate, but I noticed that from microsoftconnectivityanalyzer autodiscover only tries 443 and stops if nothing is found on that port. How can I make it try 80 if 443 fails. This was working somehow in the past

    • The client will try both HTTP and HTTPS, but generally speaking port 80 is optional as it is only used for the first step before authentication is attempted. It only responds with a 301 Redirect to ‘https://’. There are some specific examples where you may benefit, one is having lots of SIP Domains and for whatever reason you can’t add them all to the certificate – blocking 443 will allow the client to connect to http://LyncDiscover.sipdomain.co.uk/ (HTTP and ‘sipdomain.co.uk’) and will be redirected to https://webservices.primary.co.uk (HTTPS and the External Web Services FQDN which is the same regardless of the user’s SIP Domain).

Leave a Reply

Your email address will not be published. Required fields are marked *