Product reuse at Microsoft as seen through a FIM lens

Microsoft and other large software vendors often fall foul of criticisms that products overlap significantly, or that discreet functionality in one product has been written afresh when the facility is already mature in another technology. As I’ve grown to know it better, I think Microsoft’s Forefront Identity Manager (FIM) provides some interesting examples of the benefits and drawbacks of product re-use. I put these thoughts out as a set of considerations to counter the view that reuse is always a positive thing.

Note: I wrote this article a long time ago, and have always been on the fence about posting it because it’s an editorial rather than purely technical content. I’m not 100% certain this is the right place for this content, but I am publishing it here now rather than letting it rot. Because this was written a long time ago, some references are dated. Like this doesn’t speak of MIM, AADSync or AAD Connect in any detail, so put that knowledge to one side for now.

Adding capabilities to existing technologies through product reuse

Many people who know of FIM, tend to know it because it emerged within SharePoint 2010 at the heart of the User Profile Service, replacing an earlier Active Directory synchronisation component. Others may know of FIM through the Windows Azure Active Directory (WAAD) Directory Synchronisation (DirSync) tool, which is a functionally restricted version of FIM, with a Management Agent and Rules Extension that connects this FIM instance to Active Directory and Azure AD. Both of these re-uses of FIM take only a portion of the total product. We primarily interact with the Synchronisation Service Manager and the Synchronisation Service behind it (this is the older FIM stuff that’s been with the product in previous versions). I’ll talk about some of the newer stuff later in this post, but first I’ll provide my view of the success of FIM’s reuse in these technologies.

FIM in SharePoint 2010

FIM’s emergence in SharePoint 2010 was not particularly smooth. During the pre-release iterations of SharePoint 2010, FIM 2010 was approaching its own RTM. This meant that in each pre-release milestone in SharePoint 2010, FIM would change to its own new milestone release under the hood. All of this was pretty inelegant. For instance, we needed to grant the SharePoint Farm account local administration rights in order to ensure that the FIM services could provision successfully within SharePoint’s service provisioning framework. This remained the case for some time after the product reached RTM. With time and considerable attention to the specific requirements, Spencer Harbar eventually brought the community along with him to arrive at a best practice configuration of this service that would remove some of these requirements and refine the least privileges, but this service remains one of the most privilege-demanding components in SharePoint.

When SharePoint 2010 launched, the User Profile Synchronisation Service was built on a fork of the FIM 2010 Release Candidate. It still is today, even in SharePoint 2013, but this is not to say that it hasn’t benefited from significant updates over time. Indeed, I would struggle to recommend that anyone operate a younger version of it than the one found in the February 2012 Cumulative Update for SharePoint 2010. For a seasoned SharePoint practitioner, the idiosyncrasies of FIM in SharePoint are all pretty manageable, but for new initiates it can be a steep learning curve. Provisioning, re-provisioning, patching and SharePoint’s built-in backup tools all introduce specific permission changes or requirements to run with an over-privileged Farm account. Perhaps most importantly, FIM is its own beast, and I know of few SharePoint experts who have first-hand experience of FIM outside of this context. This means that troubleshooting issues within SharePoint’s instance of FIM can be problematic. As seen from an architectural ivory tower, it’s a better tool for the job than what preceded it. As an operator of SharePoint, it’s an awful lot to learn if I only want to import user data from Active Directory.

This last point clearly resonated with Microsoft. In SharePoint 2013, two new facilities have joined the User Profile Synchronisation facility I describe above. On the one hand, all of this can be offloaded to full FIM. This was always a promised technology for SharePoint 2010, but it has only recently been released for SharePoint 2013. If you already operate FIM, this may make a lot of sense, but I don’t foresee many SharePoint deployments introducing FIM just to take advantage of this added flexibility. On the other hand, Microsoft has also added a heavily optimised, less-configurable Active Directory Import mode, which clearly simplifies things, although it isn’t entirely disentangled from the SharePoint 2010 architectural changes that preceded it.

What do I make of all this? Clearly, this architectural change was motivated at least in part by two concerns.

1. Microsoft was responding to requests to broaden User Profile import and export capabilities as part of their increasing focus on Social collaboration at the time.
2. Microsoft probably wanted to eliminate the burden of maintaining a second set of code for Active Directory synchronisation when they already had a better tool for the job.

But as we can see, the reality of inserting a large chunk of one technology inside another was anything but elegant. Three years later, Microsoft found themselves in a position where they allowed full FIM to be a decoupled component, they continue to maintain the fork of FIM 2010 RC inside SharePoint, and they now also maintain a new set of code to do almost exactly what was replaced from MOSS 2007. This last investment does what the 2007 code did very well, but it now means that Microsoft has an even wider support footprint to cover. Was all of this the right thing to do in order to ship a new, more powerful social capability in 2010 rather than in 2013? That’s not such an easy question to answer by focusing purely on these questions, and I won’t succumb to that temptation here. At any rate, we begin to see some inelegance in lifting one product out of its original context in to another. While there may be tactical advantages, these may not take in to account the total impact of the decision over the life cycle of the technologies. Or all of this may be known and it may be the acceptable route to a strategic outcome.

FIM in DirSync

The Azure AD DirSync tool used to be the Microsoft Online Services DirSync tool for Office 365, and it existed in earlier incarnations for BPOS. Although my involvement with BPOS was limited, by the time I started using DirSync, it was built on Identity Lifecycle Manager (ILM) 2007 (the predecessor to FIM 2010) and this was still the case when Office 365 launched. When DirSync was upgraded to FIM 2010, it happened without much fanfare, as the FIM Synchronisation Service is largely the same as what preceded it in ILM. Most of the new FIM capabilities are outside of this service, as we’ll see later. Perhaps the most obvious change was that DirSync could be installed on 64-bit operating systems, which meant that it could run on Windows Server 2008 R2 (and later). Initially, DirSync was trailing the developments of the product that spawned it, but today it runs a reasonably current version of FIM 2010 R2.

To look at DirSync at a glance, it might be easy to get the impression that it’s a version of FIM with its wings clipped, which probably didn’t take someone very long to put together. However, having deployed the WAAD Connector for FIM, I can assure you it isn’t as simple as that. If anything, it’s a triumph that such a configurable/flexible tool can be converted in to a turn-key solution. Or if you compare it to other DirSync tools like the Yammer DSync tool (which actually just queries users for ~10 LDAP attributes), it starts to look pretty powerful for a Next > Next > Finish job.

DirSync is a critical component of successful Office 365 deployments, for administrative and usability benefits. Recent updates to it actually bring capabilities that don’t exist in the WAAD Connector for FIM, like password synchronisation, potentially saving organisations the burden of operating AD FS for single sign-on. DirSync also plays a part in the wider Azure platforms, and is a prerequisite for some services like Azure Active Directory Rights Management (AADRM). This thing is important and it isn’t going anywhere. Indeed, the new Preview version of AADSync (a multi-forest sync tool) is also built on FIM in a similar way.

Was it the right decision to use FIM for identity synchronisation to Office 365 and Azure? In my opinion, this is a resounding success. Using FIM for Active Directory synchronisation makes sense. Although troubleshooting DirSync is difficult for the uninitiated, there is now a large support knowledge base on the Office 365 Community sites, and since it has a well-known configuration, with few allowed changes, it can be supported by partners and Microsoft Support reasonably well. Lastly, Microsoft have done a very good job of giving away FIM in DirSync for free while holding back some capabilities for the full product. Namely, they don’t impose unnecessary constraints anymore. Initially, filtration of user accounts was very limited, but since these constraints have been lifted the tool has most of the power it needs, where it needs it, without giving away a lot of an expensive product for free. To sum it all up, could you imagine Office 365 without DirSync? I wouldn’t want to. But for me, this is less of an example of re-use and more indicative of re-packaging, which I will come back to in more detail later.

SharePoint in the FIM Portal

Beyond FIM’s Synchronisation Service, there are capabilities that support broader identity management objectives, like a Self-Service/administrative portal. The FIM Portal is deployed to a SharePoint Site Collection. With FIM 2010 this targeted WSS3. Over time, support for SharePoint Foundation 2010 came in to the picture, and now SharePoint Foundation 2013 is supported in SharePoint 2010 compatibility mode.

Unfortunately, the FIM Portal has no business being in SharePoint, or rather, it gets no benefits from having been built on top of it, and a raft of detriments. I say “on top of” because it’s one of the least SharePoint-y things I’ve ever seen in SharePoint. It doesn’t do anything the SharePoint way. There are SharePoint application pages behind it that don’t affect the experience in a way that a SharePoint practitioner could ever anticipate. The Search functionality has nothing to do with SharePoint Search. The workflows (a huge part of the value proposition for the portal) are custom too. The state of this portal is not horrific per se, but for someone who knows SharePoint well it is full of cognitive disconnects at every turn. It is its own thing that requires its own learning, and for a SharePoint user, significant forgetting.

From my point of view, this approach to using existing technology is a failure. It uses SharePoint as an infrastructure for a portal, but then rewrites most of the things that SharePoint provides. This violates so much of what we advocate for in the SharePoint community. In this case, SharePoint is providing the veneer of a re-purposed technology while the portal gains few of the benefits that the platform provides. SharePoint infrastructure is complicated, so it’s questionable whether any net benefits have been derived from this approach, at the expense of significant complexity, confusion and degraded user experience.

SCSM Data Warehouse in FIM Reporting

Lastly, FIM 2010 R2 introduced reporting capabilities by allowing FIM licensees to deploy a System Centre 2012 Service Manager Data Warehouse dedicated to its use within FIM. This Data Warehouse implementation has its own dependency on SQL Server Reporting Services, which has its own high availability requirements relative to other SQL Server roles. My personal experience with these relatively new components is fairly limited and I’m not experienced with SCSM outside of this context, but at face value, it’s evident that this is a capability that’s going to require scaling out the FIM farm.

Reporting is sufficiently distant from my core skills that I can’t comment well about the suitability of choosing SCSM over some other solution, or potentially building reports for SSRS directly, but I can say this is yet another complicated stack to learn when the baseline reporting capabilities may not require this level of sophistication. I suppose my gut feeling is that this capability has been provided with SCSM because it helps enhance FIM’s capabilities quickly, although it’s a pretty hefty stack to operate for such a limited use.

Re-examining the Design Principle

The reason I started down this somewhat lengthy consideration of re-use is that I often see this put forward as a design principle, or see critics cite failure to re-use technology as a prima facia failure. By focusing on how this has gone down with FIM, I find it very difficult to speak about any benefits of this approach unequivocally, even within that narrow view. I can say that it has worked well, that it has been rushed, that it has been the wrong approach, the quickest solution, or that it can look a bit over-the-top. If all of these varying assessments hold true in a wider context, it’s not a very good principle.

Typically I see this re-use principle cited with reference to Service-Oriented Architecture (SOA), making this the central principle, focusing on components and re-use from the start. That approach clearly can have massive benefits (look at AWS or Azure), but this also assumes that you have developers building stuff to tap those services. Clearly, Microsoft are shifting towards services, but they also like off-the-shelf software and SaaS. Reusing a software package is the more problematic proposition.

When considering reuse of software packages specifically, it’s important to draw a further distinction between re-packaging and re-use. If we put some restrictions on a product for discrete purposes, it’s much more likely to be successful than when a product (or chunks of a product) are subsumed in to a different product – to reside in that different context indefinitely. This is rarely going to work well, without so much rework or forking that it may be more complex/costly to re-design, re-build, fix and support the new thing throughout the remainder of its life-cycle. It would be pretty amazing if anyone could envisage the total effects of change this big.

We can see all of this in FIM. The give-away (almost “freemium”) version of the technology (DirSync) has proven successful to the point that it may have more strategic value to Microsoft than the full product. When Microsoft gives away a constrained license to a full product (SCSM Data Warehouse), enabling a new capability in FIM, it certainly limits some of the risks of reuse, but this can look bloated and distorted by virtue of this perversion of the original intent. When the SharePoint or FIM teams have lifted an entire product (or a large chunk of it) in to an existing stack, replacing or adding some integral function, things start to get painful. And this shouldn’t be surprising since the designers of the imported technology never intended this.

Ultimately, a product is not a component and it isn’t a normalised service in an SOA. We already have components in Windows and in many other places. We take advantage of that good work to make things re-usable and reduce rework all the time. This approach can scale out to distributed systems and we can build massive services that put these concerns at the heart of our process by service orientating architecture. But SOA is not the right approach for every purpose, nor for every organisation, and certainly not in the way that we conceive of off-the-shelf software today. SOA gives us a model for providing services for unknown uses. Products assemble components for use cases.

Similarly, development frameworks provide tools for quickly solving repetitious problems, but they don’t solve problems on their own. Sometimes building *anything* may be unviable. Sometimes an organisation needs a product. And a good product is unlikely to become a good part of another product. All the refinement, optimisation and (probably) de-normalisation that made the original product good will be antagonistic to its use in some new, different context. This is not to say that products should never integrate with other products or services, but that integration can get ugly when it turns in to subsumption. This is precisely the idea that I take issue with. Products do not subsume well in to other products. Whether it be through acquisition, organisational fragmentation, iterative enhancements or some other factor that leaves us with multiple tools for a single job, the right answer might be to build yet another new thing rather than lifting some existing product in to a new context that was outside of the original product designer’s intent.

Note: since originally writing this, Steven Sinofsky posted this excellent article on Apps: Shrapnel v. Bloatware, which may be of interest if you made it this far.