SharePoint 2010 Development Environment Performance: SSD, i5 vs. i7, WEI and Sandy Bridge

Late last year my colleagues and I tried to distil the tasks that impede SharePoint developer productivity. Then I ran those tests on EC2, Hyper-V and VMware Workstation, with the latter two virtualisation technologies running on a desktop, an older laptop and a newer laptop. In this post I hope to shed a bit of light on some follow-up testing that I’ve squeezed in to the odd hour here and there over the last six months. Unfortunately hardware availability and my schedule have not aligned to produce a further round of comprehensive tests and since I can’t see that occurring in the immediate future I’m going to fill in some gaps here with a couple of additional concrete findings, particularly regarding i5 vs. i7 testing and the impact of SSD on first page load times after application pool recycles. I’ll also talk less rigorously about a few related issues.

SSD

Probably the most contentious finding from my initial testing was that disk performance and bus speed aren’t significant factors in most of those results (start-up and shut-down times being a notable exception). To recap a bit of my initial summary:

Disk performance and bus speed did not prove to be significant factors in these results (except for virtual machine start-up times). Obviously there are fundamental differences about SSD (yet untested) that may skew this picture, but I will be surprised to see big differences. If we’ve got these tests right, and they are actually representative of the tasks that slow down development, then we would expect to see wider variance across bus or disk speeds. We don’t.

• This assumes the disk is relatively uncontended. Virtual machine performance degrades in every type of test while large file operations are running concurrently on the same disk. This could be copying an ISO, importing or exporting a virtual machine or any other sustained large file operations.

The obvious follow-on test would be to repeat on the same system with SSD. Unfortunately I’ve not found the time or hardware resources to do that yet, but today I ran an indicative test. In this scenario, I installed two new boots on a brand new Lenovo ThinkPad W520. One drive was an SSD in the second bay, the other was a 7200 RPM SATA drive (I don’t have specs for either to hand, but they were the default Lenovo offerings). For both boots I ran the VMs on the other spindle, so we had one test with an SSD system drive and VMs running on mechanical drive. For the second test I inverted the configuration and had a mechanical system drive with VMs running on SSD. In both cases there was no appreciable system contention outside of these tests.

The results? Identical. First page load after application pool recycle times were around 10 seconds for Central Administration, a blank site and a My Site Host. 16 seconds for a customised intranet solution (the same one from the initial tests). These are very similar times to the desktop results from my original tests – only marginally slower. What does this tell me? I should complete the testing for more scenarios than just the first page load times. But given that it won’t happen any time soon, I’m pretty comfortable assuming that SSD isn’t going to automagic performance improvements where disk speed is otherwise not a factor, and I’m happy standing by my initial analysis with this supplementary finding in hand.

To be crystal clear, I’ve seen first-hand how quickly a VM starts and shuts down when running on SSD. It’s stunning. And there’s clearly a subsequent gain reaching a post-start-up stasis of sorts. I always waited for my system to calm down like this before any testing could begin, and in some cases that might take ten minutes on a 7200 RPM mechanical drive, and even longer over USB2. However, I don’t actually see this as a major productivity loss. Irritating, yes. A sound business case? Probably less so. I imagine doing lots of full crawls would translate to a big productivity gain on SSD, but is that a major issue for most developers on most projects? Not consistently so, in my experience. But if you’re developing a FAST solution it would probably be a good idea. Maybe even isolate all of the DBs on the SSD. There would certainly be scope to play with this once you have known disk contention that you’re fighting.

The problem I have is that I can’t find any other scenarios which are as disk-bound as we might assume. When we first started this testing in late 2009, our first inclination was to add eSATA drives on a PCI Express port to get a second spindle. Freeing up the VMs from system activity and large file operations on the system disk is a clear win, but this will be true for any disk of any speed on virtually any bus if my initial test results are to be trusted, which means that the SSD investment for VM performance gains is only likely to get you faster start-up/shutdown times and anything else that involves large file operations.

All this said, if budget and SSD reliability are not concerns, load up on them, assuming it gets you sufficient storage capacity. It won’t hurt, so long as they don’t fail all the time. Additionally, it may be beneficial to get an SSD for the system drive, if other non-development activities would benefit from it. Or it may be that start-up/shutdown times are compelling on their own. In the final analysis, I’m in no way opposed to SSD, but when it’s my neck on the line for justifying hardware purchases, I want concrete, consistently-realised performance gains if I’m going to recommend a less resilient, lower capacity, more expensive technology. In most cases, I’m not sure that’s the case for virtualised SharePoint development.

i5 vs. i7

One of the other key follow-on investigations from last year’s testing was a comparison of i5 vs. i7 processors. I’ll quote the initial context here:

• The benefit of spending on i7 processors is in doubt. We are seeing very minor performance penalties when adding more than two CPUs in both VMWare Workstation and Hyper-V for most tests. There were also very minor improvements for some tasks, but on the whole there does not appear to be a measurable benefit. This might vary if the host OS is doing a great deal with the CPU, but that is liable to cause other contention issues than just in the CPU (on a laptop).
• The only tasks that appeared to use all 8 cores in a SharePoint VM were:
  • Retract/Deploy of a solution (but only very briefly)
  • Create web app, or Create site collection (but at low percentages)
  • Rebuild with Code Analysis (but not fully)

Since the initial testing, I’ve continued to experiment with two versus four cores in the VM, and have never seen a significant enough difference to endorse using more than two, but at the same time, I don’t think the penalties for multiple cores are significant enough to worry about, if any user thinks that four cores will be better. Note: I’m only talking about development here.

Based on these findings, I had a hunch that a faster clock speed i5 would outperform an i7, assuming two or fours cores running inside the SharePoint VM. For the sake of simplicity I’ve tested with two cores. For these follow-on tests I used the same ASUS V6-P7H55E model that I used during the original testing, with an identical spec/configuration and the same VM, with one exception. We replaced the Intel® Core™ i7-870 Processor (8M Cache, 2.93 GHz) with an Intel® Core™ i5-680 Processor (4M Cache, 3.60 GHz) – faster speed, smaller cache.

To my surprise, the performance tests returned virtually identical results to my initial testing (all within the margins that the initial tests deviated). Reviewing those results again, we can see that for most tests disk performance is not an issue (see above), and these tests suggest that CPU is not a bottleneck to further performance gains beyond a certain point (I believe an older CPU would fare poorly against either of these, but if a 4.0 GHz i5 came along, I’m not sure we’d see an improvement over these results). These machines have reasonably high-spec RAM, so memory speed does not seem a likely candidate for further improvements. Based on resource monitoring during testing, I can’t see that anything is maxed out, so I’m beginning to think there’s something inherently languid in the sequence of this computation. Perhaps a deeper dive is in order some day, but I’m probably not the best person to take that on.

As an aside, I can confirm that I’ve been running up to six VMs concurrently on this desktop with the i5 over the last couple of weeks. Starting all of the machines up at once is rough, but after 15-20 minutes it’s handling it no problem, and I don’t have to do that unless I’m taking a major snapshot. This suggests disk starts to become an issue with six VMs running at once, but that shouldn’t surprise anyone. If anything I’m surprised it’s not more of an issue on this machine, and if I continue to need this many VMs at once I’ll probably sacrifice my RAID 1 array for two separate disks. I’d be hesitant to suggest SSD in this case, since six VMs is probably going to chew up more storage than most SSDs will accommodate.

Based on these results and this longer-term experience, I’d recommend the higher-speed i5. I don’t seem to lose anything with the i5, at any rate. Maybe even go down to a 3.0 GHz i5 and save some money? If you know you have a specific scenario that will consistently utilise eight cores, go for the i7. But ultimately, both of these CPUs are fast.

Windows Experience Index and CPU Benchmarks

I’ve had a number of discussions with people about performance since publishing these posts, and it’s surprised me to find how many people actually look at the Windows Experience Index. Unfortunately, in my experience, this really doesn’t tell us much on today’s machines. A poor-to-average developer machine today gets a good score, unless it has a 4200 RPM hard drive (in which case it shouldn’t be used by a developer). Also, graphics performance is probably irrelevant. I really don’t think this index sheds any light on the SharePoint Development Experience Index, as it were.

Along these lines, with the receipt of a few new Sandy Bridge CPUs in these Lenovo laptops, we started running CPU benchmark tools. These are quite useful for diagnosing problems (early BIOS versions on the W520 were slooooooooooooooooooow – make sure to apply v1.25+), but beyond that, I’m not sure they tell us what we need to know for SharePoint development. For instance, at one point we saw hugely different CPU benchmark scores but the SharePoint performance tests were roughly the same. I guess I mention these tools here to say that they may be useful in some cases, but I think these real world tests probably tell us more.

Sandy Bridge

“How good are these Sandy Bridge CPUs”, I hear you mutter? Battery life is amazing. I accidentally left the Lenovo W520 running unplugged all day today. I think it lasted about six hours. Performance-wise, you’ve seen the ~10 second first page load times after an application pool recycle on a few of the standard SharePoint OOTB templates. That’s on a 2.0 GHz Sandy Bridge i7. This is not far off the 2.93 GHz first generation i7 desktop CPU results from our original tests, and much better than the first-generation i7 laptop PCU. Pretty good, I’d say. I can’t wait to see the second generation desktop speeds. Note: the desktop i7 models also have integrated graphics, where the first-generation desktop i7 CPUs did not. Now to hope there aren’t any more recall issues.

A few other things to note:

• Until a few months ago, I didn’t realise that dual-core i7 CPUs exist. I thought they were all quad-core. Not so. This is important because if you find a laptop model with four SODIMM slots (to get you 16GB RAM, and 32GB in due course), the fine print will probably tell you that you will only get two SODIMM slots unless you purchase a quad-core CPU.
  • There’s a secondary “gotcha” here, in that the quad-core laptop i7 CPUs peak at a much lower clock rate than their desktop siblings. I think the fastest first-generation quad-core laptop i7 CPU peaks at just over 2 GHz, and most laptop manufacturers have very few models, if any, with this CPU. In fact, we struggled to find anything other than 1 Lenovo, 1 Dell and 1 HP model at 15″. Most of these only had availability for lower clock speeds and we nearly had to settle for 1.73 GHz. These are all very expensive as well.
• The Sandy Bridge comes to the rescue here insofar as it has higher clock speed quad-core laptop i7 models, even if these are also slower than their desktop siblings. However, it’s also worth noting that in most CPU comparisons of Sandy Bridge to 1st-generation i7 models, the Sandy Bridge annihilates. Basically, at this point, if shopping for a high-performance SharePoint development laptop, you should be looking at Sandy Bridge. They may actually be cheaper as well – somehow.
• Also be aware that the Japanese earthquake has caused severe manufacturing delays for most hardware vendors. You may find you need to settle for a lengthy lead time at the moment.