Fundamentally, it’s irrelevant that this data was updated by these users in their MySites. The property could have been updated by an administrator in the User Profile Service Application. However, it appears that the User Profile export is not just exporting the URL as a string, it is actually copying the image on export; the User Profile Service Application is browsing to the SSL-secured site to pick up the image and writes it to the user’s thumbnailPhoto attribute. In this post I’ll review the evidence and explain the additional certificate trust configuration required to export an SSL-secured User Profile picture.

Certificate Trusts

While I initially stumbled across this error in the SharePoint ULS logs, it was also logged to the Windows Application event logs as a SharePoint Foundation Topology error (8311), “The root of the certificate chain is not a trusted root authority“. As I hinted at yesterday, I initially disregarded this error because it pertained to our wildcard SSL certificate and I knew that the root certificate (Equifax) would be trusted by Windows by default (this is pretty fundamental to a Public Key Infrastructure). It was only when I noticed that this event occurred as part of the User Profile Synchronisation process that I looked in to it in more detail. These were the ULS logs that first grabbed my attention:

08/31/2010 10:36:12.33    miiserver.exe (0x106C)    0x1A1C    SharePoint Foundation    Topology    e5mc    Medium    WcfSendRequest: RemoteAddress: 'http://<HOSTNAME>:32843/2caef145ca304d148378faec86645981/ProfileDBCacheService.svc' Channel: 'Microsoft.Office.Server.UserProfiles.IProfileDBCacheService' Action: 'http://Microsoft.Office.Server.UserProfiles/GetUserData' MessageId: 'urn:uuid:63e3bc90-878b-4fc4-99b5-38c8ab2a4574'
08/31/2010 10:36:12.34    w3wp.exe (0x0C34)    0x0E14    SharePoint Foundation    Topology    e5mb    Medium    WcfReceiveRequest: LocalAddress: 'http://<FULLY QUALIFIED HOST NAME>:32843/2caef145ca304d148378faec86645981/ProfileDBCacheService.svc' Channel: 'System.ServiceModel.Channels.ServiceChannel' Action: 'http://Microsoft.Office.Server.UserProfiles/GetUserData' MessageId: 'urn:uuid:63e3bc90-878b-4fc4-99b5-38c8ab2a4574'    9c26087b-cb49-40b2-93f5-f8583499eead
08/31/2010 10:36:12.34    w3wp.exe (0x0C34)    0x0E14    SharePoint Foundation    Monitoring    nasq    Medium    Entering monitored scope (ExecuteWcfServerOperation)    9c26087b-cb49-40b2-93f5-f8583499eead
08/31/2010 10:36:12.34    w3wp.exe (0x0C34)    0x0E14    SharePoint Foundation    Monitoring    b4ly    Medium    Leaving Monitored Scope (ExecuteWcfServerOperation). Execution Time=4.61966025838972    9c26087b-cb49-40b2-93f5-f8583499eead
08/31/2010 10:36:12.38    miiserver.exe (0x106C)    0x1A1C    SharePoint Foundation    General    erv2    Medium    Updating X.509 certificate validation policy
08/31/2010 10:36:12.44    miiserver.exe (0x106C)    0x1A1C    SharePoint Foundation    General    erv3    Medium    Adding X.509 certificate thumbprint '<THUMBPRINT 1>' to root authority trust
08/31/2010 10:36:12.44    miiserver.exe (0x106C)    0x1A1C    SharePoint Foundation    Topology    8311    Critical    An operation failed because the following certificate has validation errors:\n\nSubject Name: CN=*.<WILDCARD DOMAIN NAME>, OU=Domain Control Validated - RapidSSL(R), OU=See www.rapidssl.com/resources/cps (c)10, OU=GT12672216, O=*.<WILDCARD DOMAIN NAME>, C=GB, SERIALNUMBER= xxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxx \nIssuer Name: OU=Equifax Secure Certificate Authority, O=Equifax, C=US\nThumbprint: <THUMBPRINT 2>\n\nErrors:\n\n The root of the certificate chain is not a trusted root authority..
08/31/2010 10:36:12.44    miiserver.exe (0x106C)    0x1A1C    SharePoint Foundation    Topology    8311    Critical    An operation failed because the following certificate has validation errors:\n\nSubject Name: CN=*.<WILDCARD DOMAIN NAME>, OU=Domain Control Validated - RapidSSL(R), OU=See www.rapidssl.com/resources/cps (c)10, OU=GT12672216, O=*.<WILDCARD DOMAIN NAME>, C=GB, SERIALNUMBER=xxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxx\nIssuer Name: OU=Equifax Secure Certificate Authority, O=Equifax, C=US\nThumbprint: <THUMBPRINT 2>\n\nErrors:\n\n The root of the certificate chain is not a trusted root authority..

These certificate errors at the bottom interrupted a long series of repeated http://Microsoft.Office.Server.UserProfiles/GetUserData events. The two events at the top of this excerpt are the last in that series, so the certificate errors occurred immediately after the user data had been gathered. This was a pretty strong hint that the certificate errors were interfering with the export. I was initially pretty stumped because I knew Windows trusted this certificate and I could confirm the certificates were fully trusted when I browsed to the MySite from the server, but I didn’t know what requested the certificate and why it wasn’t trusted. So I searched around a bit and found a number of articles about adding root certificates to SharePoint 2010′s Manage Trusts, which I’ve only used for Claims and Cross-Farm Service Applications, but I noticed it also came up for FAST Search, Exchange Web Services and REST – seemingly for any web services that access SSL-secured resources, intra or inter-farm.

What I don’t understand is why SharePoint web services aren’t using the Windows certificate store as a first stop before using SharePoint’s own Manage Trusts. I’m assuming this is something to do with Claims but I’m struggling to uncover anything meaningful that can confirm or disconfirm this. In the absence of guidance, I tested exporting our wildcard certificate and the root CA’s certificate to the local machine’s and the farm account’s certificate stores to see if that would fix the problem, without success. Next I tried to import the wildcard certificate through Manage Trusts, which still did not fix the certificate errors, but when I imported the root CA’s certificate in to Manage Trusts, the next synchronisation succeeded. In an effort to get to grips with these requirements and to understand how the certificate was requested I continued to monitor in ULSViewer and the FIM Client.

Successful Export

Initially, the same two GetUserData actions ran as in the first two events above. In fact, each of the first five ULS entries were identical to the first excert above, followed by the successful addition of three certificates to the root authority trust. This was when I suspected things were working (or that I’d at least cleared the next hurdle).

08/31/2010 12:22:10.65    miiserver.exe (0x07DC)    0x1B1C    SharePoint Foundation    General    erv3    Medium    Adding X.509 certificate thumbprint '<THUMBPRINT 2>' to root authority trust
08/31/2010 12:22:10.67    miiserver.exe (0x07DC)    0x1B1C    SharePoint Foundation    General    erv3    Medium    Adding X.509 certificate thumbprint '<THUMBPRINT 3>' to root authority trust
08/31/2010 12:22:10.72    miiserver.exe (0x07DC)    0x1B1C    SharePoint Foundation    General    erv3    Medium    Adding X.509 certificate thumbprint '<THUMBPRINT 1>' to root authority trust

At this point a number of “Assemblies and Sequences” are registered. Pretty boring stuff, so omitted here. All of these events begin with SPDelegateManager or SPXmlConfigurationManager. This is followed by a few events verifying the upgrade status of the databases, confirming that upgrades are not in progress or necessary. After that there are a couple of handfuls of additional configuration checks. Continuing after all that, we see the expected HTTP GET request for the first user’s updated User Profile picture:

08/31/2010 12:22:14.61    w3wp.exe (0x1B08)    0x09EC    SharePoint Foundation    Monitoring    nasq    Medium    Entering monitored scope (Request (GET:https://<MY SITE HOST>:443/User%20Photos/Profile%20Pictures/<USER 1 PROFILE PICTURE NAME>.jpg))
08/31/2010 12:22:14.61    w3wp.exe (0x1B08)    0x09EC    SharePoint Foundation    Logging Correlation Data    xmnv    Medium    Name=Request (GET: https://<MY SITE HOST>:443/User%20Photos/Profile%20Pictures/<USER 1 PROFILE PICTURE NAME>.jpg))    b7ca5cb5-25f0-401d-87e8-fe93bb3c63d5 

This is followed by a few less relevant events, then more log entries about the GET:

08/31/2010 12:22:14.93    w3wp.exe (0x1B08)    0x09EC    SharePoint Foundation    Monitoring    b4ly    Medium    Leaving Monitored Scope (Request (GET: https://<MY SITE HOST>:443/User%20Photos/Profile%20Pictures/<USER 1 PROFILE PICTURE NAME>.jpg))). Execution Time=325.341137549723    b7ca5cb5-25f0-401d-87e8-fe93bb3c63d5
08/31/2010 12:22:14.93    w3wp.exe (0x1B08)    0x09EC    SharePoint Foundation    Monitoring    nasq    Medium    Entering monitored scope (Request (GET: https://<MY SITE HOST>:443/User%20Photos/Profile%20Pictures/<USER 1 PROFILE PICTURE NAME>.jpg)))
08/31/2010 12:22:14.93    w3wp.exe (0x1B08)    0x09EC    SharePoint Foundation    Logging Correlation Data    xmnv    Medium    Name=Request (GET: https://<MY SITE HOST>:443/User%20Photos/Profile%20Pictures/<USER 1 PROFILE PICTURE NAME>.jpg))    dfaaa0df-b2c0-4aab-a461-71e39680ad08
08/31/2010 12:22:14.95    w3wp.exe (0x1B08)    0x09EC    SharePoint Foundation    Monitoring    b4ly    Medium    Leaving Monitored Scope (Request (GET: https://<MY SITE HOST>:443/User%20Photos/Profile%20Pictures/<USER 1 PROFILE PICTURE NAME>.jpg))). Execution Time=3.98852138030115    dfaaa0df-b2c0-4aab-a461-71e39680ad08
08/31/2010 12:22:14.95    w3wp.exe (0x1B08)    0x09EC    SharePoint Foundation    Monitoring    nasq    Medium    Entering monitored scope (Request (GET: https://<MY SITE HOST>:443/User%20Photos/Profile%20Pictures/<USER 1 PROFILE PICTURE NAME>.jpg)))
08/31/2010 12:22:14.95    w3wp.exe (0x1B08)    0x09EC    SharePoint Foundation    Logging Correlation Data    xmnv    Medium    Name=Request (GET: https://<MY SITE HOST>:443/User%20Photos/Profile%20Pictures/<USER 1 PROFILE PICTURE NAME>.jpg))    d3a5dff3-8dad-4c79-abf1-66284748d246 

A few more irrelevant events, then:

08/31/2010 12:22:14.98    w3wp.exe (0x1B08)    0x17EC    SharePoint Foundation    General    af74    Medium    HTTP request URL: /User%20Photos/Profile%20Pictures/<USER 1 PROFILE PICTURE NAME>.jpg)    d3a5dff3-8dad-4c79-abf1-66284748d246 

More database upgrade checks, then:

08/31/2010 12:22:16.18    w3wp.exe (0x1B08)    0x09EC    SharePoint Foundation    Monitoring    b4ly    Medium    Leaving Monitored Scope (Request (GET: https://<MY SITE HOST>:443/User%20Photos/Profile%20Pictures/<USER 1 PROFILE PICTURE NAME>.jpg)). Execution Time=1240.43874963807    d3a5dff3-8dad-4c79-abf1-66284748d246

These events are then repeated sequentially for other updated users. After these complete, voila! The thumbnailPhoto attribute is exported by FIM, populated in Active Directory and the profile picture appears in the Outlook 2010 Social Connector. I haven’t monitored how quickly this appears, but it works without further intervention.

And the money shot. Who wouldn’t want to see this mug in their Outlook every day. :/

Summary

Following this interrogation, it’s clear to me the certificate errors come from the location of the updated profile picture. The errors can be fixed by importing an export of the root CA’s certificate through SharePoint’s Manage Trusts (possibly the site’s certificate itself as well, although the error was on the root trust). That SharePoint 2010 web services (including the User Profile Service Application) do not appear to use the Windows certificate store feels uncomfortable to me, but I imagine this is probably related to the new SharePoint 2010 STS and I will be looking at this topic in more detail in future.

We initially followed Matthew McDermott’s Profile Image Export suggestions but in our case these steps were insufficient, as detailed below. That article was written while SharePoint 2010 was a beta product. The User Profile Service Application changed since that release and is now configured differently, so it doesn’t surprise me that our experience differs.

You might wonder why we spent this much effort just to get a picture in Active Directory (of all places). While we think it’s important to have this knowledge for our clients and delegating photo selection to end users can drive SharePoint adoption, it is also used by the Outlook 2010 Social Connector. When you start using this great new social computing front-end, it just feels incomplete without a photo.

Identifying the Trouble

We’ve recently deployed a new SharePoint 2010 farm for team sites, but we wanted to make certain that the User Profile Service Application was fully configured before unleashing it, as it’s not the sort of thing that you want to add to a system that’s already in production. In our initial testing we completed a full profile import then decided to test exporting changes to a few of the Active Directory properties, such as mobile number and the profile picture, or thumbnailPhoto (as it’s known in LDAP). This is done by removing the Import mapping in the User Profile Property and adding a new setting for Export (see Matthew McDermott’s blog above if this is unclear). The mobile number synchronised successfully but the thumbnailPhoto failed, with no obvious errors.

After running a couple of full synchronisations while actively monitoring ULSViewer, I also launched the FIM (Forefront Identity Manager) Client, AKA Synchronisation Service Manager (or MIISClient.exe), as Spencer Harbar recommends in his Rational Guide. Note that although this is not a supported tool for SharePoint 2010, it provides a view of what’s going on in SharePoint’s FIM instance that isn’t always visible in the ULS logs. This can sometimes be invaluable.

Stepping through these FIM events, it was clear that the LDAP thumbnailPhoto property (PictureURL in SharePoint) wasn’t getting marked for export. We could monitor the addition of the Mobile number when that data was updated, but the PictureURL was not captured.

Active Directory Rights

Revisiting the source, I decided to see if the TechNet documentation included anything specific for the User Profile picture, as it was last updated on 12 August 2010 and I’d not looked at it since May. Happily it speaks specifically to these requirements, albeit in rather cryptic terms:

To export properties, such as profile pictures, from SharePoint Server 2010 to AD DS, at least Replicate Directory Changes permission is needed on the object and all child objects for the AD DS domains to which you want to export data from SharePoint Server 2010. Read/Write permission is also needed on the container that stores the user picture attribute, for example, the ThumbnailPhoto attribute.

Unfortunately my Active Directory skills are not as sharp as they were when I actually  administered a domain, but I figured it shouldn’t be rocket science to figure out how to grant permissions on an attribute. Pretty much everything I found mentioned granting read/write permissions on the following user attributes:

Read/Write – jpegPhoto
Read/Write – pwdLastSet
Read/Write – userAccountControl

However, I still didn’t know how to do that. Luckily the fifth post in this thread spells it out:

1. Right-click the appropriate OU
2. Select the Security tab
3. Click Advanced
4. Click Add
5. Select user then OK
6. Select Properties tab
7. Change Apply To to Descendant User Objects
8. Check both Read and Write for the following permissions:
thumbnailPhoto
userAccountControl
pwdLastSet

Note: I tried to set the permissions on thumbnailPhoto without including the other two attributes and the sync failed in precisely the same manner as it had previously.

After granting these rights in my development domain and kicking off a User Profile sync I saw the PictureURL attribute appear in FIM and it successfully updated Active Directory. However, when I repeated this process in our production environment I still had the same behaviour as before. At this point I had another look through the ULS logs in painstaking detail and this time I paid attention to two certificate errors (more on that tomorrow). But before pursuing any certificate troubleshooting missions I made a note and hit the search engines one last time in anger.

SQL Native Client

In my searches I also found that FIM 2010 requires the SQL 2008 Native Client on installations where SQL is installed on another server. My development environment has local SQL but production uses a shared instance. I tried to run the installer for the Native Client in production but was notified that it already existed. On reflection, I remembered this is installed by the SharePoint 2010 pre-requisite installer, which explains why this issue is being reported by full FIM users and not by SharePoint 2010 users. However, it’s worth noting this requirement here in case someone has an overzealous administrator that decides to remove the Native Client.

SSL

It will turn out that my production environment requires additional work because the MySite web application is SSL-secured. I will go over those requirements in my next post, although it should not be necessary to follow those steps unless the User Profile Service Application is reading data from an SSL-secured site.

The machine-default permission settings do not grant Local Activation permission for the COM Server application with CLSID
{000C101C-0000-0000-C000-000000000046}
and APPID
{000C101C-0000-0000-C000-000000000046}
to the user <FARM ACCOUNT> SID (S-1-5-21-xxxxxxx….) from address LocalHost (Using LRPC). This security permission can be modified using the Component Services administrative tool.

The CLSID for this COM Server Application is MSIServer, used to activate the Windows Installer Service. You can find this by navigating to HKCR\AppId and examining the details there:

Given that there were 105 instances of this DCOM 10016 error in an eleven second period, I decided to see what was happening at the same time (00:52:09-00:52:19) in the Application event logs. It turned out that there were 210 Information and Warning events during the same time-frame. An example pair of these event is included here:

1035 Information
Windows Installer reconfigured the product. Product Name: Microsoft Excel Mobile Viewer Components. Product Version: 14.0.4763.1000. Product Language: 0. Manufacturer: Microsoft Corporation. Reconfiguration success or error status: 0.

1015 Warning
Failed to connect to server. Error: 0×80070005

You might notice this first informational event is for the Microsoft Excel Mobile Viewer. As you trawl through the events it will become clear that these events occur for Language Packs, Service Applications, Foundation elements, Web Apps – all sorts. At this point it was pretty clear to me that the SharePoint Farm account (probably a Timer Job) was trying to run the Windows Installer Service for these products, but I had no idea why, so I cracked open ULS Viewer and went to town.

A Timer Job called job-admin-product-version kicked off at 00:52:08.92. Filtering by that CorrelationID I could see that this job appeared to build a hierarchy of upgradable elements (Foundation stuff, Service Applications, etc), then checked to see if these elements can be upgraded. But I don’t really feel comfortable trying to figure out what a Timer Job does by stepping through ULS logs, so before going any further I had a look at the TechNet Timer Job Reference and found that Product Version Job runs nightly at 00:45 by default.

Now that we know that this job “checks the install state of the machine and puts that data into the database“, I’m going to take a leap of faith and assume that the farm account is trying to use the Windows Installer Service to do this. Hopping back in to the ULS logs, the next-to-last event correlates to the deluge of Application logs entries.

Note that one of the first items in this mess of updates is Microsoft Excel Mobile Viewer Components again.

UpdateProductInfoInDatabase, regProductsQuery = exec proc_RegisterProductVersion N’20c667df-1bc3-486b-869c-a3ba40f83af5′, N’Microsoft SharePoint Server 2010′, N’14.0.4763.1000′, N’{90140000-1138-0000-1000-0000000FF1CE}’, N’Microsoft Excel Mobile Viewer Components’, N”…

It seems pretty clear that Product Version Job checks the installed versions of SharePoint Products and Technologies, then updates that info in the database (presumably Central Admin config). However, it’s not really clear when that information gets used, so changes to the default Daily job schedule may have unintended consequences. As I see it, this leaves three options:

• Live with the warnings/errors until a better option becomes available.
• Disable the Product Version Job timer job, noting that this could potentially have a negative impact on updates to the system (not recommended until these implications are better understood).
  • Potentially combine this strategy with a plan to make the farm account a local admin temporarily and run the job manually at routine intervals (again, this warrants testing and a better understanding of the Timer Job itself).
• Grant the WSS_ADMIN_WPG local group or Farm account permissions to Launch and Activate the Windows Installer Service (which I don’t recommend).

I’m presently contending with these errors in a development environment, so I’m going to live with them for now. I’m pretty reluctant to recommend the last option. It seems to me that if the Farm account has rights to elevate to Local System via the Windows Installer Service, that puts a pretty big dent in the least-privileged model. I’ll keep looking in to this, but I thought I’d identify my findings so far and I would welcome any comments or ideas that I haven’t considered – particularly if anyone has more information about when this database information is used.

I hit the search engines in earnest and found that these problems were prevalent across a fairly wide range of graphics cards. We enlisted Dell’s help and they told us that they do not certify that Hyper-V will work on any laptops. More precisely, they clarified the primary support concern is that future driver releases may not work with Hyper-V even if we find a model that works with today’s drivers. At this point we were considering a pricier Precision model and they put us in touch with their Precision product team in Texas. They were most helpful but we were told that Dell themselves do not use Hyper-V on laptops except for demonstration purposes and they simply use it as a server for connected workstations, so they would never experience the same graphics issues. Dell kindly offered to let us test our development build on various models at their campus if we agreed to share the results with them, but before we could arrange that visit, Windows Server 2008 R2 SP1 Beta* was released and I upgraded my machine in order to test out Dynamic Memory.

As I was installing it I had a chat with my colleague (and serial early adopter) Lambros Vasiliou to gauge his impressions. He mentioned his favourite improvement is that the known Hyper-V host graphics performance issues are either gone or greatly mitigated. This is an issue that’s been repeatedly discussed in our organisation since we moved from a hotchpotch of virtualisation technologies to Hyper-V as our standard development build last year. It’s probably the single thing that irritates our users of this system more than anything else.

I did some testing myself with videos playing and moving windows about with Windows Key + Arrow hot keys. The results were fairly impressive – without doubt a big improvement. One thing that still behaved poorly on my Dell XPS M1330 (with NVIDIA GeForce 8400GS) is full-screen YouTube, Vimeo, etc. The CTRL+ALT+DEL redraw operation seems a bit sluggish still as well. I noticed that my PowerPoint Presenter View was better, but still not 100% responsive.

I also tested on the Dell Latitude E6410 (with NVIDIA NVS 3100M). Not only is the previously-mentioned blue screen fixed and the graphics generally improved in the same ways as on the XPS, but the full-screen in-browser video and CTRL+ALT+DEL are instantaneous. One possible explanation for this different experience is that the Latitude has a processor with SLAT, but I can’t validate that at all yet… because I can’t find any information whatsoever about why/how this has changed!

I think it’s unlikely that these changes are related to RemoteFX (since the XPS M1330 does not have a processor with SLAT and I never enabled it on the Latitude E6410). I would expect RemoteFX to improve the experience connecting to the guests, not the Hyper-V root partition (although it’s possible that this improvement is somehow related). I’ve tried pinging Virtual PC Guy and posted this query on the SP1 Beta TechNet forum but so far the community can only confirm that this is indeed working on a number of different models including a Mac (drill down in the links on the TechNet thread for more information). One way or the other this is great news, but I’m finding the lack of information about these changes quite maddening given the amazing detail that’s been produced for the Dynamic Memory launch. I’d really appreciate further insights if anyone can reveal the internals.

* A few notes regarding the Service Pack 1 Beta installation process:

1. The links on the SP1 Beta page are a bit confusing. You should be aware that if you click the “Evaluate Windows Server 2008 R2 and SP1 Beta” link you will be taken to a page with a “Download Windows Server 2008 R2 Trial Software” section at the top. “Download SP1 Beta Software” is beneath that section. This is what you want. If you click the first link you will initiate a download of the full Windows Server 2008 R2 (SP0) installer. If you “upgrade” your system using that installer you’ll wind up with a nice new trial version of SP0. AGH. Starting again from the links in the right section I was able to run a small installer that presents the updates to Windows Update and that has all worked fine, so I’d recommend that route. Alternately the Service Pack can be downloaded stand-alone. I did that for my second install and it worked fine too. Also note the Windows Server 2008 R2 SP1 Beta Reviewer’s Guide, “to evaluate the core features of Windows Server 2008 R2 SP1 Beta release in your environment”.
2. If you use Forefront you will need to uninstall it in order to install SP1 Beta, so make sure to remember to reinstall it afterwards.
3. When I installed the Service Pack my screen went black for about ten minutes following the first reboot. Be prepared for this. You’ll see plenty of ongoing disk activity but nothing on the screen. Fairly disconcerting, but perhaps this is all a part of these same video changes.

Exchange Server Reconfiguration

Tidying up the Exchange server is a much more straight-forward process. In fact, all of the changes that I made are network orientated per the network changes from the first post, so if you are not adding a second NIC or a second fixed IP address on the original internal NIC, these steps aren’t necessary.

• Import the virtual machine. Plug the Hyper-V Internal Network in to the first NIC and add a second NIC with the Hyper-V ICS Network plugged in to it.
• The initial IP address is 192.168.150.2.
• Rename the Local Area Connection NIC to Hyper-V Internal Connection (or your preference).
• Rename the Local Area Connection 2 (or maybe 3) NIC to Hyper-V ICS Connection (or your preference).
• In IPv4 properties on the Hyper-V ICS Connection, change the Advanced TCP/IP Settings to not “Register this connection’s addresses in DNS”, as it is dynamic.
• I disabled IPv6 on both NICs as it is already disable on our host’s network connections.
• Add a second IPv4 address to the Hyper-V Internal Connection:  192.168.200.151/255.255.255.0.
  • Added the same address on the DNS tab for this NIC.

• Check DNS to make sure the new address was added.
• Added a HOSTS file entry for demo2010b pointing at 192.168.200.151.
• Tested logging on via Remote Desktop from the Host machine.

Shutdown/snapshot and optionally export the VM if time/resources permit.

Reviewing SharePoint Server event logs

This section details my review of the event logs in the shipped state. I did not take any action except for the last two items regarding DCOM fixes and the SharePoint Health Logs. I believe that most of these errors are probably an effect of running so many things in one environment, but I’d welcome comments if you have any insight to share.

• SetSPN for WSMAN warnings: The WinRM service failed to create the following SPNs: WSMAN/demo2010a.contoso.com; WSMAN/demo2010a.
  Additional Data
  The error received was 8344: %%8344.
  User Action
  The SPNs can be created by an administrator using setspn.exe utility.
  Presumably these SPNs can be created manually or the rights to create the SPNs can be assigned to the WinRM service account if needed, but I am not making any changes here until I see that it is necessary to do so. There are some Kerberos Audit Failures in the security logs but since the SharePoint environment is self-contained and there are no secondary hops, I don’t think this is worthwhile.
• VSS Error 8320: Volume Shadow Copy Service error: Failed resolving account Administrator with status 1376. Check connection to domain controller and VssAccessControl registry key.
  Operation:
  Initializing Writer
  Context:
  Writer Class Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
  Writer Name: WMI Writer
  Error-specific details:
  Error: NetLocalGroupGetMemebers(Administrator), 0×80070560, The specified local group does not exist.
  There are a number of articles that discuss fixes for this Warning but since this is an unusual configuration (SharePoint/SQL on a DC) and the warning is about the “Administrator” account, I am hesitant to make this change, for fear of introducing instability. Further reading at Experts Exchange, Technet and AnandTech.
• There are two errors regarding the ULS config file, 7105 and 7056. I’m honestly unsure what to make of these errors since they refer to C:\Program Files\Common Files\Microsoft Shared\ULS\14\uls.config.xml and I’m not certain how that is used in generating the ULS log files for SharePoint which are appearing normally at C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\LOGS. Not taking any action for now.
• Microsoft.ResourceManagement.ServiceHealthSource Error 22:
  The Forefront Identity Manager Service cannot connect to the SQL Database Server.
  The SQL Server could not be contacted. The connection failure may be due to a network failure, firewall configuration error, or other connection issue. Additionally, the SQL Server connection information could be configured incorrectly.
  Verify that the SQL Server is reachable from the Forefront Identity Manager Service computer. Ensure that SQL Server is running, that the network connection is active, and that the firewall is configured properly. Last, verify the connection information has been configured properly. This configuration is stored in the Windows Registry.
  This error seems to occurs before the FIM Sync service starts, which may or may not be a related issue. The User Profile Service Application is available though, so I believe this has something to do with this connection attempt occurring before the Service Application is able to connect with SQL.
• CAPI2 Error:
  Failed extract of third-party root list from auto update cab at: <http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootstl.cab> with error: A required certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file.
  This issue seems to have emerged in July of this year on Windows 7 and Windows Server 2008 R2 machines. Unfortunately there’s no clear resolution at this time, but also no clear negative impact on this system, so I’m not taking any action for now.
• DCOM 10016 error. A fix for this has been documented well by Matt Groves.
• SharePoint Health Reports:
  • Almost all of the warnings and errors in the Health Report are either one-time issues that no longer exist or are symptoms of the single-server install on a DC.
  • An exception to this is the “Validate the My Site Host and individual My Sites are on a dedicated Web application and separate URL domain” message, which is accurate. The environment has not been configured with a dedicated MySite application.
  • The one other issue was “The Unattended Service Account Application ID is not specified or has an invalid value”. This has not been set up in the VM and will need to be configured in the Secure Store Service if it will be used.
  • I have chosen to disable all of these rules, as they are undesirable in a demo environment. I also deleted the existing alerts.

Pending

As I mentioned above, I hope to revisit this VM and create new snapshots that will reduce load by disabling bulky services. This snapshot branch might resemble something like this:

• Current state, as above
  • Most SharePoint Services turned off in Services on Server
  • BI Indexing Connector Service , FAST, OCS and Project Server removed. All SharePoint Services turned on
  • As above, but with FAST
  • As above, but with OCS
  • As above, but with Project Server
  • BI Indexing Connector Service , FAST, OCS and Project Server removed. All SharePoint Services turned off
  • As above, but with Search/FAST
  • As above, but with OCS
  • As above, but with Project Server

I really haven’t planned this yet or discussed these options with users of this environment, so it could wind up looking completely different. Also, this many snapshots would probably chew up too much disk space and get confusing for the users. But watch this space for updates as I hope to revisit the topic again. And please feel free to suggest other optimisations or additions that work well for you.

Now the public beta trial is expiring and people are moving to the RTM build. It appears to be much improved, in that more of the product works in this version and a few niggles have been fixed now. However, it’s widely acknowledged that the resource requirements for this virtual machine are gargantuan due to the breadth of what it offers.

What you get

The first of these VMs doesn’t just have SharePoint Server 2010 (Enterprise). It has:

• Active Directory Domain Services
• DNS
• SQL 2008 R2
• Office Professional Plus 2010 (including Visio and Project)
• SharePoint Designer 2010
• Visual Studio 2010
• Office Web Applications
• FAST Search for SharePoint 2010
• Project Server 2010
• Office Communication Server 2007 R2

There is a second VM with Exchange Server 2010 which you can use as needed. Note: when downloading the RAR files which self-extract to the VM export files, I highly recommend using the Akamai download links. They will save you time and frustration.

Why I’m writing about it

The first of these VMs does a lot, and will chew up more virtual resources than most physical demo machines have to provide. In this post I documented the tweaks I made to our copy of the Virtual Machines and I’ve reviewed the event logs to identify any other issues that are present in the shipped state. I also spent some time fixing the start-up of some services due to observed delays in network connectivity.

The optimisations achieved here are not immense but they establish a firmer baseline from which to carry out further improvements. Unfortunately I have not had the time to explore the best approach to taking these core assets and removing the bulky bits in order to provide snapshots that unleash bits of the functionality as needed. With time I hope to revisit this and potentially explore other uses for this environment.

SharePoint Server Reconfiguration

These are the steps I’ve taken to reconfigure the SharePoint Virtual Machine:

• Imported the virtual machine. Plugged the Hyper-V Internal Network in to the first NIC and added a second NIC with the Hyper-V ICS Network plugged in to it. To understand more about the NIC naming conventions I’m using here and how we use these Hyper-V networks, please refer to my series on SharePoint development environments.
  • If you’re curious, we dual-boot our Sales laptops in to this development environment so our sales people can use Hyper-V as needed.
• The default SharePoint VM RAM is set to 5120MB (5GB) RAM. This could be increased up to ~6GB on a host with 8GB RAM, assuming the load on the root partition is minimal during demonstration and the Exchange VM is not being used concurrently.
• If using a UK keyboard through the Hyper-V Virtual Machine Connection, the password will need to be entered as pass“word1
• Note: at login a warm-up script runs, which I will discuss more in a moment.
• I changed the time zone to London.
• I changed UAC to “Notify me only when programs try to make change to my computer (do not dim my desktop)”. I find that with Known Hyper-V graphics performance issues this setting achieves the right balance between usability and security.

Service start-up

• I reviewed the event logs to identify which Manual Services would start up at a delay. Approximately five minutes after the machine started the Virtual Disk Service started up. That was followed by the following services:

  Service	Startup Type
  Application Experience	Manual
  BI Indexing Connector Service	Automatic (Delayed Start)
  Certificate Propagation	Manual
  Diagnostic Policy Service	Automatic (Delayed Start)
  Diagnostic System Host	Manual
  Distributed Transaction Coordinator	Automatic (Delayed Start)
  FAST Search for SharePoint	Automatic
  FAST Search for SharePoint Browser Engine	Manual
  FAST Search for SharePoint QRProxy	Manual
  FAST Search for SharePoint Sam Admin	Manual
  FAST Search for SharePoint Sam Worker	Manual
  Function Discovery Provider Host	Manual
  IPsec Policy Agent	Manual
  Microsoft .Net Framework NGEN v4.0.30319_X86	Automatic (Delayed Start)
  Microsoft .Net Framework NGEN v4.0.30319_X64	Automatic (Delayed Start)
  Network Connections	Manual
  Network List Service	Manual
  Office 64 Source Engine	Manual
  Office Software Protection Platform	Manual
  Portable Device Enumerator Service	Manual
  Remote Access Connection Manager	Manual
  Remote Desktop Services	Manual
  Remote Desktop Services Configuration	Manual
  Remote Desktop Services UserMode Port Redirector	Manual
  Secure Socket Tunneling Protocol Service	Manual
  SPP Notification Service	Manual
  SQL Full-Text Daemon Launcher	Manual
  Telephony	Manual
  Windows Defender	Automatic (Delayed Start)
  Windows Module Installer	Manual
  Windows Remote Management	Automatic (Delayed Start)
  Windows Search	Automatic (Delayed Start)
  Windows Update	Automatic (Delayed Start)
  WinHTTP Web Proxy Auto-Discovery Service	Manual

• In my testing, switching the Network Connections and WinHTTP Web Proxy Auto-Discovery Service services to Automatic would reduce time to CTRL+ALT+DEL from five minutes to three minutes. It’s worth verifying these results in your environment.

Service Hardening

I disable the following services, per my normal virtual machine hardening processes. I don’t believe that any of them are necessary in this environment:

• Certificate Propagation
• Desktop Windows Manager Session Manager
• Distributed Link Tracking Client
• Encrypted File System
• Function Discovery Provider Host
• Function Discovery Resource Publication
• IP Helper
• Microsoft iSCSI Initiator Service
• Multimedia Class Scheduler
• Problem Reports and Solutions Control Panel Support
• Remote Procedure Call (RPC) Locator
• Smart Card
• Smart Card Removal Policy
• Special Administration Console Helper
• Tablet PC Input Service
• Windows Audio
• Windows Audio Endpoint Builder
• Windows Error Reporting Service
• Windows Search
• Wired AutoConfig

I also disabled these services. I have no idea why they were ever enabled.

• Microsoft Fibre Channel Platform Registration Service
• Microsoft FTP Service

Following a reboot to assess boot times after hardening, I was prompted with a firewall exception warning for SharePoint Workspace. I allowed this exception.

MSCONFIG

I mentioned earlier that a warm-up script launched at login. I reviewed the contents of the script and identified that it enumerated all sites in the farm, so I browsed to each site that was getting warmed up to have a look at that content. Most of these sites pull up the “select a template” page, indicating that there is not content in the web application, so I have chosen to disable the script for now and will likely revisit the use of the IIS.NET application warm-up module when it matures. In the mean time, I recommend that recipients of this build take a snapshot of the VM while the http://intranet and Central Admin web applications are fully warmed up for their needs so they can roll back to that point as needed.

While in MCSONFIG, I turned off:

• Watson Subscriber for SENS Network Notifications
• EcmCopyLinks
• Warm Up
• Optional:
  • Microsoft Office 2010 (these are sync services – probably best to leave them running)
  • Microsoft Office Communicator 2007 R2 (this is preferential)

Housekeeping and simple benchmarks

At this point I wanted to take some basic start-up timings so I shut down the VM, but before starting back up I changed the Hyper-V VM’s BIOS settings to boot from IDE before CD. After that I took some pretty unscientific measurements of boot times and resource consumption on my Dell XPS M1330:

• CTRL+ALT+DEL: 2:56.
• Desktop: 3:47.
• Baseline RAM consumption down from >4.5GB to ~3.65GB before hitting the intranet and Central Administration web apps (remember, they aren’t warmed up anymore).
• RAM consumption at just under 4GB after hitting Central Administration.
• RAM consumption at ~4.25GB after hitting http://intranet.

After taking these measurements I shut down the VM and took a snapshot.

Network changes

These changes are optional, depending on how you will use the VM in your environment. I opted to add a second IP address on our existing internal network range, as follows:

• The initial IP address is 192.168.150.1/255.255.255.0.
• Rename the Local Area Connection NIC to Hyper-V Internal Connection (or your preference).
• Rename the Local Area Connection 2 (or maybe 3) NIC to Hyper-V ICS Connection (or your preference).
• In IPv4 properties on the Hyper-V ICS Connection, change the Advanced TCP/IP Settings to not “Register this connection’s addresses in DNS”, as it is dynamic.
• I disabled IPv6 on both NICs as it is already disable on our host’s network connections.
• Add a second IPv4 address to the Hyper-V Internal Connection: 192.168.200.150/255.255.255.0.
  • I also added 192.168.200.150 on the DNS tab for this NIC.
• Check DNS to make sure the new address is added.
• Add a HOSTS file entry in the root partition for demo2010a pointing at 192.168.200.150.
  • If desirable, add another entry pointing for intranet.contoso.com pointing at 192.168.200.150. This can be used for browsing from the root partition’s browser.
• Tested logging on via Remote Desktop and browsing to Central Admin from the Host machine.

Client tools and other changes

Some final changes are optional, but I think generally desirable.

• Install Firefox, Opera, Safari, Chrome.
• Install PDF Exchange Viewer or your choice of PDF viewer.
• Add the environment variable path to the 14 hive’s BIN.

And that’s it for the SharePoint VM. Shutdown and take a new snapshot, optionally deleting the first one. I would suggest exporting the VM if time/resources permit, noting that export operations can be time-consuming and disk intensive.

In part two I’ll discuss the Exchange VM, event logs and potential future improvements.

To fix this just requires a couple of lines of PowerShell, courtesy of this article (to which I’ve added some clarification here).

If you go to the Manage Service Applications link you’ll see the problem:

Open the SharePoint 2010 Management Shell and Run as Administrator.

Run Get-SPServiceApplicationProxy to enumerate the IDs of all the Service Application Proxies in your farm.

(note: you will probably have more Service Application Proxies than these)

Copy the ID for the WSS_UsageApplication.

Run the following two lines of PowerShell.

$UP = Get-SPServiceApplicationProxy | where {$_.ID -eq "<PASTE COPIED ID HERE>"}
$UP.Provision()

If you refresh the Manage Service Application page the proxy should be started now.

Once that has been configured, the usage data will not appear immediately in all of the usage reports because the timer jobs to collect the data will not have run yet. The granularity of the data collection and processing is pretty fine if you want to reconfigure them for your needs, but keep in mind that none of the Web Analytics reports will appear for at least 24 hours after this fix is in place. They do not allow reporting on the current day. If you try to configure a Custom Date Range to include today’s date you will get the following message:

I’ll try to track this issue as updates come out.

Installing the SEO Toolkit

Although Remote Server Administration Tools can be installed on Windows Vista and Windows 7, I have produced the directions below on my Windows Server 2008 R2 desktop. The instructions should be fundamentally the same for any OS once IIS Manager is available locally, however it is installed. To be crystal clear, the SEO Toolkit can be used by anyone with Windows Vista, Windows 7, Windows Server 2008 or Windows Server 2008 R2. It is not a requirement to have access to the web server and it is not necessary to install IIS locally.

On Windows Server 2008 and 2008 R2 the IIS Manager Feature can be added through Server Manager, even if the IIS Server Role is not installed.

Expand the Remote Server Administration Tools node and select the Web Server (IIS) Tools node.

Click Next and Install. Wait for the Add Features Wizard to complete and then download and install the SEO Toolkit. IIS Manager is available in the Administrative Tools menu and should look something like this when you click on your local machine’s connection in the left-hand pane.

The SEO Toolkit home page looks like this.

I shan’t go over everything here because there is an excellent three minute video on the SEO Toolkit home page (linked above), which details the basic functionality of the tool.

Analysing a SharePoint 2010 Publishing site

Click the first link on the SEO Toolkit landing page and Create a new analysis of your site.

The analysis takes you directly to the Site Analysis Report.

Clearly, the Violations are of interest. What sort of things do they tell us? I’ll look at the errors first.

Drilling in to The title is missing, we find that the five pages are links to authenticated content on the out of the box Publishing site template’s home page, which are 401 unauthenticated errors in this case, since this is an anonymous access zone.

The Publishing Portal home page, which includes links to authenticated content

The individual violations

So these errors are unsurprising, now that we know what they are. If this were real content, ideally we would modify it to remove these links to authenticated pages.

So what about that Canonical Formats message?

This violation tells us that a single object can be accessed using two different links. In this case we have two sets of duplicated images. The first two .png files are transparent spacers and the second two are orange.

That’s precisely the sort of thing we would hope to find out about and correct. So what about “The page contains broken hyperlinks”?

Again, all of these links are broken because they point to authenticated content. It’s the same story for “The URL for the hyperlink is broken”, except for the five .gif files that appear there.

In this case the images are actually missing. Again, this is exactly the sort of thing we want to know.

Unsurprisingly all of these errors point out fundamental problems in the site which content owners would want to correct even if they were unconcerned with page rank. The warnings, which I skipped over earlier, provide supplementary insight in to changes that can improve page rank in an otherwise functionally-correct website. Rather than discussing those individually, I’ll just include a screen shot of the Violations tab in the bottom half of the details pane. This tab summarises all of the violations on a selected page, which will improve the editor’s experience when making these changes.

What’s particularly useful about this view is that it is now enumerating each page that violates the specified rule in the top pane, but the Violations tab enumerates all of the violations for the selected object in the top pane (the default home page in this instance).

Some of the other dashboards reveal slow-performing pages, most linked pages, redirects, pages blocked by robots.txt, a status code summary, a list of external links and more. It’s a very useful set of tools. If this has been at all interesting, it’s definitely worth reviewing the video and other resources up at IIS.NET.

Server tools

Everything that I’ve discussed so far can be run against any site that the SEO Toolkit user can browse to. Server access is completely unnecessary. However, there are two added tools that have to be run on the server. This should not be hugely problematic for the content owner, as these tools need to be updated relatively infrequently once they’ve been set up initially.

Creating a Sitemap

Hopping on to my server, I click the Create a new sitemap link and specify the Sitemap file name.

This is my favourite bit. When the Add URLs dialogue first launches, it displays the IIS site files in inetpub.

Not very useful for a SharePoint site, is it? But if we Run new Site Analysis from the URL structure drop-down…

The New Analysis box pops up.

After about 40 seconds of analysis (in my environment) we get the SharePoint site map!

At the bottom of the dialogue the Change Frequency tells search engines how often pages are likely to change. The priority details how important we consider our site to be relative to other URLs on our site (I must confess I don’t fully understand how this works, but I’m not responsible for content. ). You can also tell Search Engines how to identify the Last Modified Date. More information on all of this can be found on TechNet.

One last thing before moving on. We need to add the Sitemap to our robots.txt file. There’s a handy link in the Actions pane to do so.

Also note, in the Related Features on the Actions pane, there’s a link to Robots Exclusion, which brings us to the final tool.

Robots.txt file management

If we click the Robots Exclusion link or the View existing rules link from the SEO Toolkit landing page, we can see that our sitemap.xml file is being referenced, as added above.

This is confirmed if I click the Open Robots.txt link in the Actions pane.

If we click Add Disallow Rules or Add Allow Rules we get a similar dialogue, and in both cases we will want to specify our previous Site Analysis for the URL structure.

Now if I want to exclude all of the content that I don’t want to index I can just tick the appropriate boxes from my last analysis (note that this looks different than the selectable options from the Sitemap dialogue).

Voila! Paths are excluded and the robots.txt file is updated.

One thing to note is that the robots.txt file did not appear for me during testing until after I stopped and started the website in IIS. This is only an issue when it’s created for the first time, but worth noting. I believe this is also true for the sitemap.xml. file.

So… these are good tools! Once configured, the robots.txt shouldn’t need to be updated often and the web managers should become aware of any problems soon after they occur through their own use of the reporting tool. In short, these tools devolve a great deal of control and insight and there seems to be very little reason not to use them.

We have also experimented with generating the server side outputs using PowerShell, which a colleague of mine will detail soon and I will post here when ready. If there is any reluctance to use this IIS.NET extension in production infrastructure, a combination of PowerShell for file generation/management and the SEO Toolkit for reporting may be a sensible solution.

• Microsoft’s approach to Dynamic Memory is fundamentally different than VMWare’s overcommitment, in that VMWare doesn’t trust information about memory usage from within the guest, whereas Microsoft’s implementation is based around an awareness of the amount and type of memory that’s being used at all times.
• Dynamic Memory will work by Adding/Removing memory.
  • Adding memory is enabled through a new synthetic memory driver.
  • Removing memory that’s not being used is done with ballooning.
  • Memory is now assigned with a few new values:
    • Startup memory is the amount of memory assigned to a VM, which is also the minimum memory the VM will consume (default value is 512 MB).
    • Maximum memory limits how much memory a VM can consume.
    • Priority can be assigned to specific VMs in order to make sure that they receive available memory before other lower-priority VMs.
    • A Memory Buffer can be set to reserve memory for specific VMs, for instance if they need extra memory for file caching.
• Hyper-V Manager adds two new columns.
  • Current Memory identifies how much memory the VM is consuming.
  • Memory Availability identifies the difference between how much memory a VM has vs. wants in a +/-% figure.
    • When the availability goes negative, the Windows guest will start to work with the lesser amount of memory that’s now available to it (via paging, etc).
    • Negative availability will result in reduced performance, but the systems will continue to function.
• Memory is now reserved for the root partition in a different way, so that dynamic memory won’t bring down the host.
  • This amount can be configured with a new registry key based on how the root partition is being used, for instance if it’s your desktop OS.
• As Dynamic Memory is used more, the chances of spanning NUMA nodes increases (on NUMA systems).
  • He points out that different systems have vastly different Back Channel performance, so the impact of NUMA Spanning can be negligible or drastic.
  • In SP1, NUMA Spanning can be disabled (if desired).
• Dynamic Memory also supports Large Pages, which are likely to become more common with virtualised Exchange/SQL.
  • VMWare cannot overcommit these pages.
• I’ve asked if there are specific processor requirements. I’ll be interested to see how/if this supports processors that don’t have SLAT.