My colleague Mike Parker has a great new series of posts up on securing Exchange Server 2016 with Azure AD. This option may seem counter-intuitive at a glance, but given that most organisations are on a trajectory from Exchange Server to Exchange Online, this configuration can consolidate access control for e-mail in a single location (for instance, over the duration of a migration or for long-term co-existence). It also means that Azure AD Conditional Access policies can be used for Exchange resources on and off-premises, which improves security while enabling mobility.
This configuration has two parts:
- Get most Exchange Server components to use OAuth 2.0. This is known as Hybrid Modern Authentication.
- Publish Outlook Web App (OWA) and the Exchange Control Panel (ECP) using the Azure AD Application Proxy.
The second step is necessary because these components are not currently supported for Hybrid Modern Authentication. The major pre-requisite for publishing an application with the Azure AD Application Proxy is that it should be authenticated with Kerberos and the Application Proxy Connector machine accounts need to be configured to use Constrained Delegation (KCD) for the OWA and ECP Service Principal Name (SPN). Mike’s article takes you through all of this step-by-step. My post deviates a bit from Mike’s guide to consider the idiosyncrasies of the Exchange Alternative Service Account Credential (ASA), which has underpinned Kerberos in Exchange Server since 2010 SP1. If you are familiar with configuring Kerberos, the ASA will almost certainly hold some surprises. Maybe even a fourth head.
Mike’s guide (above) is important because a lot of the existing community content for Exchange Server and the Application Proxy leaves some gaps for you to fill, for instance, if your Exchange infrastructure consists of more than one server. So when it came time to apply the Kerberos configuration, and it didn’t work (because invariably something gets overlooked when you configure Kerberos in a new context for the first time), Mike, Andrew Bogard and I put our heads together to troubleshoot, which required getting to grips with the ASA.
- First up, we had a look at KList on the OWA client machine. Not getting a ticket. Kerberos blatantly isn’t working.
- Next, we checked the Exchange service accounts and the Application Pool Identities on the two CAS servers. Nothing seems to be running under the service account that has been configured as the “Alternative Service Account” (ASA) Credential. We’ll come back to this.
- Next, we confirmed the URL was added to the Local Intranet zone. I could write another post about misunderstanding how the, “Include all local (intranet) sites not listed in other zones” option works, but someone beat me to it. Hint: explicitly add the URL to the Local Intranet Zone. I assume it is normal for OWA and ECP to share the same URL, but if not, add both.
- Next, we pinned everything to one server. Exchange can use round-robin DNS, which is unusual for most authenticated web technologies, and I wanted to make sure this wasn’t causing any problems. So we pinned DNS to one node to focus the troubleshooting.
- Finally, we tried to interrogate the security logs on this server, but there were no Kerberos failure events for the OWA requests. We were receiving an Integrated Windows Authentication logon prompt, so we knew IIS was trying to use Kerberos, but not succeeding. We couldn’t even correlate any logon events to the ASA Credential.
At this point I was convinced something was going wrong inside Exchange when the ASA was configured, and I was adamant that Kerberos wouldn’t work unless the Application Pool Identities, or possibly an Exchange Service account was actually running as the ASA. I could see no evidence of that whatsoever, so Kerberos can’t work, right? RIGHT? But eventually, after kicking all of these doors repeatedly, it started working, and the Application Proxy connection followed suit. Clearly everything must be working if the Application Proxies could successfully delegate. But how!?!?! The Service Accounts and Application Pool Identities all still run under machine accounts.
It turns out I’m not the first person with this confusion. Thankfully, this thread points to the original article about the inner workings of the Alternative Service Account (ASA) credential from when it was introduced in Exchange 2010 SP1. Frustratingly, these internals are no longer a part of the current ASA documentation. I use the term “internals” fairly loosely, as the ASA is effectively a black box.
The crux of this original documentation is that the “Client Access server service host” (I believe this is the Microsoft Exchange Service Host service that is running on a CAS server) has been extended with a “servicelet”. This paragraph clears up a few things about how Exchange is atypical, but doesn’t really unveil the mechanics in detail.
“The Client Access server service host has been extended to use a shared credential for Kerberos authentication. This service host extension monitors the local machine. When credentials are added or removed, the Kerberos authentication package on the local system and the network service context is updated. As soon as a credential is added to the authentication package, all client access services can use it for Kerberos authentication. The Client Access server will also be able to authenticate service requests addressed directly in addition to being able to use the shared credential. This extension, known as a servicelet, runs by default and requires no configuration or action to run.”
I don’t understand how this design decision got through. Extending the Kerberos authentication package specifically for two accounts, rather than running services or application pools under a different account (like every other service under the sun), seems extremely… hacked. And things that are hacked with proprietary code can’t be effectively investigated when things go wrong. Ultimately, things have been this way for a long time now, and they are unlikely to change, but I don’t think it’s great that Kerberos in Exchange has a fundamental reliance on proprietary code, in a way that most other Microsoft technologies do not.
I am guessing the way the black box works is something like this.
- IIS receives the Kerberos authentication request.
- The IIS application pool identity runs under either the Local System or Network Service machine account.
- The local machine account will use the Kerberos Authentication package to handle the Kerberos request, and the Kerberos Authentication Provider has been extended in a way that allows these two machine accounts to decrypt Kerberos tickets for the ASA once the machine account receives them back from the Domain Controller.
This is all speculation (what else can we do with a black box?), so I’d be keen to hear if you interpret this differently. I would guess that these machine accounts are either decrypting tickets with the ASA’s stored password hash using the DPAPI (normal Windows stuff), or they are given the hash when the credential is set. If it’s the latter, it would be good to understand better how it is stored. One article suggests it is stored in the registry, which could be a vague way of saying it uses the DPAPI, or it could be a different mechanism.
To try to clarify my gripe, I understand that having a Kerberos extension for specific uses has precedence, and I am not suggesting that Windows shouldn’t be extended. What I find weird is that the extension appears to be for specific accounts, rather than system-wide uses, and there is no clear reason why Exchange is such a special case. We are changing the behaviour of authentication for the Operating System as a whole because one service has chosen to change the system rather than work within it.
The PKU2U Negotiate extension for HomeGroup is an example of how I would expect to see an authentication package extension work. Windows is fundamentally extensible. It is one of the primary design principles of Windows, dating back to NT. This is all good. Introducing an extension without explaining why the normal ways of doing things don’t apply, is not good. And leaving the original rationale behind in older documentation (as vague as it was) certainly doesn’t help.
So we are left with an unjustified and undocumented black box that makes troubleshooting harder. You will not see security events logged as the ASA, so it is not a “service account” in the way that anyone understands it. Perhaps there is a way to access logs for the black box, but who would know if this was the case? I’m hoping there are some other troves of documentation that some kind soul will point me at, but I’m not holding my breath. In the meantime, I hope that resurfacing these idiosyncrasies will help people who know Kerberos through these hiccoughs at a time when Kerberos might become more popular for Exchange.