Product Version Job: DCOM 10016 strikes again

For some time now, IT professionals have been modifying DCOM activation rights in order to keep their System event logs clean. In SharePoint 2010, that fix became slightly trickier, as permissions to modify the DCOM permissions had to be granted through the registry for the IIS WAM REG admin service and oSearch14 DCOM applications. Having made these fixes, I’ve noticed a new breed of DCOM 10016 error.

The machine-default permission settings do not grant Local Activation permission for the COM Server application with CLSID
{000C101C-0000-0000-C000-000000000046}
and APPID
{000C101C-0000-0000-C000-000000000046}
to the user <FARM ACCOUNT> SID (S-1-5-21-xxxxxxx….) from address LocalHost (Using LRPC). This security permission can be modified using the Component Services administrative tool.

The CLSID for this COM Server Application is MSIServer, used to activate the Windows Installer Service. You can find this by navigating to HKCRAppId and examining the details there:

Given that there were 105 instances of this DCOM 10016 error in an eleven second period, I decided to see what was happening at the same time (00:52:09-00:52:19) in the Application event logs. It turned out that there were 210 Information and Warning events during the same time-frame. An example pair of these event is included here:

1035 Information
Windows Installer reconfigured the product. Product Name: Microsoft Excel Mobile Viewer Components. Product Version: 14.0.4763.1000. Product Language: 0. Manufacturer: Microsoft Corporation. Reconfiguration success or error status: 0.

1015 Warning
Failed to connect to server. Error: 0x80070005

You might notice this first informational event is for the Microsoft Excel Mobile Viewer. As you trawl through the events it will become clear that these events occur for Language Packs, Service Applications, Foundation elements, Web Apps – all sorts. At this point it was pretty clear to me that the SharePoint Farm account (probably a Timer Job) was trying to run the Windows Installer Service for these products, but I had no idea why, so I cracked open ULS Viewer and went to town.

A Timer Job called job-admin-product-version kicked off at 00:52:08.92. Filtering by that CorrelationID I could see that this job appeared to build a hierarchy of upgradable elements (Foundation stuff, Service Applications, etc), then checked to see if these elements can be upgraded. But I don’t really feel comfortable trying to figure out what a Timer Job does by stepping through ULS logs, so before going any further I had a look at the TechNet Timer Job Reference and found that Product Version Job runs nightly at 00:45 by default.

Now that we know that this job “checks the install state of the machine and puts that data into the database“, I’m going to take a leap of faith and assume that the farm account is trying to use the Windows Installer Service to do this. Hopping back in to the ULS logs, the next-to-last event correlates to the deluge of Application logs entries.

Note that one of the first items in this mess of updates is Microsoft Excel Mobile Viewer Components again.

UpdateProductInfoInDatabase, regProductsQuery = exec proc_RegisterProductVersion N’20c667df-1bc3-486b-869c-a3ba40f83af5′, N’Microsoft SharePoint Server 2010′, N’14.0.4763.1000′, N'{90140000-1138-0000-1000-0000000FF1CE}’, N’Microsoft Excel Mobile Viewer Components’, N”…

It seems pretty clear that Product Version Job checks the installed versions of SharePoint Products and Technologies, then updates that info in the database (presumably Central Admin config). However, it’s not really clear when that information gets used, so changes to the default Daily job schedule may have unintended consequences. As I see it, this leaves three options:

• Live with the warnings/errors until a better option becomes available.
• Disable the Product Version Job timer job, noting that this could potentially have a negative impact on updates to the system (not recommended until these implications are better understood).
  • Potentially combine this strategy with a plan to make the farm account a local admin temporarily and run the job manually at routine intervals (again, this warrants testing and a better understanding of the Timer Job itself).
• Grant the WSS_ADMIN_WPG local group or Farm account permissions to Launch and Activate the Windows Installer Service (which I don’t recommend).

I’m presently contending with these errors in a development environment, so I’m going to live with them for now. I’m pretty reluctant to recommend the last option. It seems to me that if the Farm account has rights to elevate to Local System via the Windows Installer Service, that puts a pretty big dent in the least-privileged model. I’ll keep looking in to this, but I thought I’d identify my findings so far and I would welcome any comments or ideas that I haven’t considered – particularly if anyone has more information about when this database information is used.

Update: 20/2/2011

I’ve done some more work on this now.

Inside Manage Patch Status

Testing Manage Patch Status