Product Version Job: DCOM 10016 strikes again

For some time now, IT professionals have been modifying DCOM activation rights in order to keep their System event logs clean. In SharePoint 2010, that fix became slightly trickier, as permissions to modify the DCOM permissions had to be granted through the registry for the IIS WAM REG admin service and oSearch14 DCOM applications. Having made these fixes, I’ve noticed a new breed of DCOM 10016 error.

The machine-default permission settings do not grant Local Activation permission for the COM Server application with CLSID
{000C101C-0000-0000-C000-000000000046}
and APPID
{000C101C-0000-0000-C000-000000000046}
to the user <FARM ACCOUNT> SID (S-1-5-21-xxxxxxx….) from address LocalHost (Using LRPC). This security permission can be modified using the Component Services administrative tool.

The CLSID for this COM Server Application is MSIServer, used to activate the Windows Installer Service. You can find this by navigating to HKCRAppId and examining the details there:

Given that there were 105 instances of this DCOM 10016 error in an eleven second period, I decided to see what was happening at the same time (00:52:09-00:52:19) in the Application event logs. It turned out that there were 210 Information and Warning events during the same time-frame. An example pair of these event is included here:

1035 Information
Windows Installer reconfigured the product. Product Name: Microsoft Excel Mobile Viewer Components. Product Version: 14.0.4763.1000. Product Language: 0. Manufacturer: Microsoft Corporation. Reconfiguration success or error status: 0.

1015 Warning
Failed to connect to server. Error: 0x80070005

You might notice this first informational event is for the Microsoft Excel Mobile Viewer. As you trawl through the events it will become clear that these events occur for Language Packs, Service Applications, Foundation elements, Web Apps – all sorts. At this point it was pretty clear to me that the SharePoint Farm account (probably a Timer Job) was trying to run the Windows Installer Service for these products, but I had no idea why, so I cracked open ULS Viewer and went to town.

A Timer Job called job-admin-product-version kicked off at 00:52:08.92. Filtering by that CorrelationID I could see that this job appeared to build a hierarchy of upgradable elements (Foundation stuff, Service Applications, etc), then checked to see if these elements can be upgraded. But I don’t really feel comfortable trying to figure out what a Timer Job does by stepping through ULS logs, so before going any further I had a look at the TechNet Timer Job Reference and found that Product Version Job runs nightly at 00:45 by default.

Now that we know that this job “checks the install state of the machine and puts that data into the database“, I’m going to take a leap of faith and assume that the farm account is trying to use the Windows Installer Service to do this. Hopping back in to the ULS logs, the next-to-last event correlates to the deluge of Application logs entries.

Note that one of the first items in this mess of updates is Microsoft Excel Mobile Viewer Components again.

UpdateProductInfoInDatabase, regProductsQuery = exec proc_RegisterProductVersion N’20c667df-1bc3-486b-869c-a3ba40f83af5′, N’Microsoft SharePoint Server 2010′, N’14.0.4763.1000′, N'{90140000-1138-0000-1000-0000000FF1CE}’, N’Microsoft Excel Mobile Viewer Components’, N”…

It seems pretty clear that Product Version Job checks the installed versions of SharePoint Products and Technologies, then updates that info in the database (presumably Central Admin config). However, it’s not really clear when that information gets used, so changes to the default Daily job schedule may have unintended consequences. As I see it, this leaves three options:

  • Live with the warnings/errors until a better option becomes available.
  • Disable the Product Version Job timer job, noting that this could potentially have a negative impact on updates to the system (not recommended until these implications are better understood).
    • Potentially combine this strategy with a plan to make the farm account a local admin temporarily and run the job manually at routine intervals (again, this warrants testing and a better understanding of the Timer Job itself).
  • Grant the WSS_ADMIN_WPG local group or Farm account permissions to Launch and Activate the Windows Installer Service (which I don’t recommend).

I’m presently contending with these errors in a development environment, so I’m going to live with them for now. I’m pretty reluctant to recommend the last option. It seems to me that if the Farm account has rights to elevate to Local System via the Windows Installer Service, that puts a pretty big dent in the least-privileged model. I’ll keep looking in to this, but I thought I’d identify my findings so far and I would welcome any comments or ideas that I haven’t considered – particularly if anyone has more information about when this database information is used.

Update: 20/2/2011

I’ve done some more work on this now.

Inside Manage Patch Status

Testing Manage Patch Status

25 thoughts on “Product Version Job: DCOM 10016 strikes again”

  1. Just found your article after working out that the Product Version Timer job was the culprit. Have you discovered any further information yet regarding a solution?

    Cheers

    David

    1. No luck yet David. I keep seeing traffic coming in from various TechNet threads – most of which wind up referencing my article in the end. I haven’t had any suggestions about other things to try and in all of the other threads it just seems to get people scratching their heads. I’m reluctantly considering that this job might have been poorly designed from a security perspective. Unfortunately I’m not sure when I’ll find the time to do any testing and in retrospect, I don’t think the tests would be exhaustive enough for me to firmly recommend any actions without knowing more from Microsoft. I may see if I can explore other angles in the community next year. I’m on paternity leave until mid-January, so it will be a few weeks before I can pick this back up properly. If you find anything yourself, I’d love to know what you come up with.

  2. We tried to give the farm account Local Activation permissions on the MSIServer ({000C101C-0000-0000-C000-000000000046}).
    The DCOM error messages in the system log disappeared, but the warnings in the application logs still appears.
    Maybe some folder access has to be given?

    1. Hey Martin,

      I’ll have an updated post on this topic in the next few days. I’ve been looking at it pretty closely. By they way, I wouldn’t recommend keeping those rights in place.

      Cheers,

      Tristan

    2. Hi Martin,

      I’ve done some more work on this now, which you may find useful:

      http://tristanwatkins.com/index.php/inside-manage-patch-status/

      http://tristanwatkins.com/index.php/testing-manage-patch-status/

      I’m afraid I never got to the bottom of why the warnings appear, regardless of the DCOM permissions. The one thing I did notice is that they didn’t appear when the Product Configuration Wizard runs one of the same methods (detailed above). I assume this must be because something requires admin rights and the Setup account has those rights. In short: not sure what to make of this, and it seems to happen within the calls to the Windows Installer Service, which I’m definitely not an expert in.

      Cheers,

      Tristan

  3. Hi,

    I’m an engineer working for Microsoft and wanted to give you information about this issue.
    It is a known issue, unfortunately without an official KB article. For the 2010 version, there is intention to fix it (though it was meant to be fixed with the upcoming SP2) other than everyone of you is submitting a ticket to give a priority on this. However, it will be fixed for 2013 eventually and will be fixed in the next version.

    The given workaround, to add the account that is running the OWSTIMER service to the local admin group, is the only proper workaround.

    Hope this helps 🙂

    1. Hi Robin. Thanks for taking the time to comment. I appreciate that it’s above and beyond.

      When you say admin rights are the only proper workaround, do you mean that’s the only way to a) get the job to properly function, b) get rid of the DCOM 10016 errors or c) get rid of the MSIInstaller warnings? I was under the impression that granting the DCOM rights would clear the error and get the job working but that there was no fix for the MSIInstaller without granting admin rights. Is that correct?

          1. It appears that this problem never got fixed. We are running SP2013 with the latest CU and still get this issue

Leave a Reply

Your email address will not be published.