Late last year my colleagues and I tried to distil the tasks that impede SharePoint developer productivity. Then I ran those tests on EC2, Hyper-V and VMware Workstation, with the latter two virtualisation technologies running on a desktop, an older laptop and a newer laptop. In this post I hope to shed a bit of light on some follow-up testing that I’ve squeezed in to the odd hour here and there over the last six months. Unfortunately hardware availability and my schedule have not aligned to produce a further round of comprehensive tests and since I can’t see that occurring in the immediate future I’m going to fill in some gaps here with a couple of additional concrete findings, particularly regarding i5 vs. i7 testing and the impact of SSD on first page load times after application pool recycles. I’ll also talk less rigorously about a few related issues.
Probably the most contentious finding from my initial testing was that disk performance and bus speed aren’t significant factors in most of those results (start-up and shut-down times being a notable exception). To recap a bit of my initial summary:
Disk performance and bus speed did not prove to be significant factors in these results (except for virtual machine start-up times). Obviously there are fundamental differences about SSD (yet untested) that may skew this picture, but I will be surprised to see big differences. If we’ve got these tests right, and they are actually representative of the tasks that slow down development, then we would expect to see wider variance across bus or disk speeds. We don’t.
- This assumes the disk is relatively uncontended. Virtual machine performance degrades in every type of test while large file operations are running concurrently on the same disk. This could be copying an ISO, importing or exporting a virtual machine or any other sustained large file operations.
The obvious follow-on test would be to repeat on the same system with SSD. Unfortunately I’ve not found the time or hardware resources to do that yet, but today I ran an indicative test. In this scenario, I installed two new boots on a brand new Lenovo ThinkPad W520. One drive was an SSD in the second bay, the other was a 7200 RPM SATA drive (I don’t have specs for either to hand, but they were the default Lenovo offerings). For both boots I ran the VMs on the other spindle, so we had one test with an SSD system drive and VMs running on mechanical drive. For the second test I inverted the configuration and had a mechanical system drive with VMs running on SSD. In both cases there was no appreciable system contention outside of these tests.
The results? Identical. First page load after application pool recycle times were around 10 seconds for Central Administration, a blank site and a My Site Host. 16 seconds for a customised intranet solution (the same one from the initial tests). These are very similar times to the desktop results from my original tests – only marginally slower. What does this tell me? I should complete the testing for more scenarios than just the first page load times. But given that it won’t happen any time soon, I’m pretty comfortable assuming that SSD isn’t going to automagic performance improvements where disk speed is otherwise not a factor, and I’m happy standing by my initial analysis with this supplementary finding in hand.
To be crystal clear, I’ve seen first-hand how quickly a VM starts and shuts down when running on SSD. It’s stunning. And there’s clearly a subsequent gain reaching a post-start-up stasis of sorts. I always waited for my system to calm down like this before any testing could begin, and in some cases that might take ten minutes on a 7200 RPM mechanical drive, and even longer over USB2. However, I don’t actually see this as a major productivity loss. Irritating, yes. A sound business case? Probably less so. I imagine doing lots of full crawls would translate to a big productivity gain on SSD, but is that a major issue for most developers on most projects? Not consistently so, in my experience. But if you’re developing a FAST solution it would probably be a good idea. Maybe even isolate all of the DBs on the SSD. There would certainly be scope to play with this once you have known disk contention that you’re fighting.
The problem I have is that I can’t find any other scenarios which are as disk-bound as we might assume. When we first started this testing in late 2009, our first inclination was to add eSATA drives on a PCI Express port to get a second spindle. Freeing up the VMs from system activity and large file operations on the system disk is a clear win, but this will be true for any disk of any speed on virtually any bus if my initial test results are to be trusted, which means that the SSD investment for VM performance gains is only likely to get you faster start-up/shutdown times and anything else that involves large file operations.
All this said, if budget and SSD reliability are not concerns, load up on them, assuming it gets you sufficient storage capacity. It won’t hurt, so long as they don’t fail all the time. Additionally, it may be beneficial to get an SSD for the system drive, if other non-development activities would benefit from it. Or it may be that start-up/shutdown times are compelling on their own. In the final analysis, I’m in no way opposed to SSD, but when it’s my neck on the line for justifying hardware purchases, I want concrete, consistently-realised performance gains if I’m going to recommend a less resilient, lower capacity, more expensive technology. In most cases, I’m not sure that’s the case for virtualised SharePoint development.
i5 vs. i7
One of the other key follow-on investigations from last year’s testing was a comparison of i5 vs. i7 processors. I’ll quote the initial context here:
- The benefit of spending on i7 processors is in doubt. We are seeing very minor performance penalties when adding more than two CPUs in both VMWare Workstation and Hyper-V for most tests. There were also very minor improvements for some tasks, but on the whole there does not appear to be a measurable benefit. This might vary if the host OS is doing a great deal with the CPU, but that is liable to cause other contention issues than just in the CPU (on a laptop).
- The only tasks that appeared to use all 8 cores in a SharePoint VM were:
- Retract/Deploy of a solution (but only very briefly)
- Create web app, or Create site collection (but at low percentages)
- Rebuild with Code Analysis (but not fully)
Since the initial testing, I’ve continued to experiment with two versus four cores in the VM, and have never seen a significant enough difference to endorse using more than two, but at the same time, I don’t think the penalties for multiple cores are significant enough to worry about, if any user thinks that four cores will be better. Note: I’m only talking about development here.
Based on these findings, I had a hunch that a faster clock speed i5 would outperform an i7, assuming two or fours cores running inside the SharePoint VM. For the sake of simplicity I’ve tested with two cores. For these follow-on tests I used the same ASUS V6-P7H55E model that I used during the original testing, with an identical spec/configuration and the same VM, with one exception. We replaced the Intel® Core™ i7-870 Processor (8M Cache, 2.93 GHz) with an Intel® Core™ i5-680 Processor (4M Cache, 3.60 GHz) – faster speed, smaller cache.
To my surprise, the performance tests returned virtually identical results to my initial testing (all within the margins that the initial tests deviated). Reviewing those results again, we can see that for most tests disk performance is not an issue (see above), and these tests suggest that CPU is not a bottleneck to further performance gains beyond a certain point (I believe an older CPU would fare poorly against either of these, but if a 4.0 GHz i5 came along, I’m not sure we’d see an improvement over these results). These machines have reasonably high-spec RAM, so memory speed does not seem a likely candidate for further improvements. Based on resource monitoring during testing, I can’t see that anything is maxed out, so I’m beginning to think there’s something inherently languid in the sequence of this computation. Perhaps a deeper dive is in order some day, but I’m probably not the best person to take that on.
As an aside, I can confirm that I’ve been running up to six VMs concurrently on this desktop with the i5 over the last couple of weeks. Starting all of the machines up at once is rough, but after 15-20 minutes it’s handling it no problem, and I don’t have to do that unless I’m taking a major snapshot. This suggests disk starts to become an issue with six VMs running at once, but that shouldn’t surprise anyone. If anything I’m surprised it’s not more of an issue on this machine, and if I continue to need this many VMs at once I’ll probably sacrifice my RAID 1 array for two separate disks. I’d be hesitant to suggest SSD in this case, since six VMs is probably going to chew up more storage than most SSDs will accommodate.
Based on these results and this longer-term experience, I’d recommend the higher-speed i5. I don’t seem to lose anything with the i5, at any rate. Maybe even go down to a 3.0 GHz i5 and save some money? If you know you have a specific scenario that will consistently utilise eight cores, go for the i7. But ultimately, both of these CPUs are fast.
Windows Experience Index and CPU Benchmarks
I’ve had a number of discussions with people about performance since publishing these posts, and it’s surprised me to find how many people actually look at the Windows Experience Index. Unfortunately, in my experience, this really doesn’t tell us much on today’s machines. A poor-to-average developer machine today gets a good score, unless it has a 4200 RPM hard drive (in which case it shouldn’t be used by a developer). Also, graphics performance is probably irrelevant. I really don’t think this index sheds any light on the SharePoint Development Experience Index, as it were.
Along these lines, with the receipt of a few new Sandy Bridge CPUs in these Lenovo laptops, we started running CPU benchmark tools. These are quite useful for diagnosing problems (early BIOS versions on the W520 were slooooooooooooooooooow – make sure to apply v1.25+), but beyond that, I’m not sure they tell us what we need to know for SharePoint development. For instance, at one point we saw hugely different CPU benchmark scores but the SharePoint performance tests were roughly the same. I guess I mention these tools here to say that they may be useful in some cases, but I think these real world tests probably tell us more.
“How good are these Sandy Bridge CPUs”, I hear you mutter? Battery life is amazing. I accidentally left the Lenovo W520 running unplugged all day today. I think it lasted about six hours. Performance-wise, you’ve seen the ~10 second first page load times after an application pool recycle on a few of the standard SharePoint OOTB templates. That’s on a 2.0 GHz Sandy Bridge i7. This is not far off the 2.93 GHz first generation i7 desktop CPU results from our original tests, and much better than the first-generation i7 laptop PCU. Pretty good, I’d say. I can’t wait to see the second generation desktop speeds. Note: the desktop i7 models also have integrated graphics, where the first-generation desktop i7 CPUs did not. Now to hope there aren’t any more recall issues.
A few other things to note:
- Until a few months ago, I didn’t realise that dual-core i7 CPUs exist. I thought they were all quad-core. Not so. This is important because if you find a laptop model with four SODIMM slots (to get you 16GB RAM, and 32GB in due course), the fine print will probably tell you that you will only get two SODIMM slots unless you purchase a quad-core CPU.
- There’s a secondary “gotcha” here, in that the quad-core laptop i7 CPUs peak at a much lower clock rate than their desktop siblings. I think the fastest first-generation quad-core laptop i7 CPU peaks at just over 2 GHz, and most laptop manufacturers have very few models, if any, with this CPU. In fact, we struggled to find anything other than 1 Lenovo, 1 Dell and 1 HP model at 15″. Most of these only had availability for lower clock speeds and we nearly had to settle for 1.73 GHz. These are all very expensive as well.
- The Sandy Bridge comes to the rescue here insofar as it has higher clock speed quad-core laptop i7 models, even if these are also slower than their desktop siblings. However, it’s also worth noting that in most CPU comparisons of Sandy Bridge to 1st-generation i7 models, the Sandy Bridge annihilates. Basically, at this point, if shopping for a high-performance SharePoint development laptop, you should be looking at Sandy Bridge. They may actually be cheaper as well – somehow.
- Also be aware that the Japanese earthquake has caused severe manufacturing delays for most hardware vendors. You may find you need to settle for a lengthy lead time at the moment.
7 thoughts on “SharePoint 2010 Development Environment Performance: SSD, i5 vs. i7, WEI and Sandy Bridge”
Very interesting read, thanks.
I just bought a 16GB quad core i7 W520 (Sandy Bridge) and new Crucial M4 SSD and dev seems faster, even from my previous laptop which was a 1st gen dual core i5, 8GB RAM and slightly slower SSD.
Hard drives seem to be the bottle neck for me; I could get away with less RAM for a single VM dev environment but I want a small farm.
What do you know about SSD alignment? Apparently with SSDs the partition starting offset (run msinfo32) needs to be divisible by 4096 but I don’t know much about this.. apparently if it is not divisible there are significant performance penalties. As far as I know this should usually be the case if Windows 7 was used for partitioning.
Hi Gus. I’m not familiar with SSD alignment but I do recall reading about how disks are carved up and that the block size is fairly large. I also recall that a lot of the penalties are incurred over time – so you will get great performance initially and then (depending on the capability of the drive) it will perform worse as it gets more fragmented. IIRC this was mostly to do with random write access, but unfortunately I don’t have the detail to-hand. This was mostly from a really good article in PC Pro magazine (I think it was) a couple of months ago, which a colleague loaned me, but unfortunately it’s not available on soft copy. I came to the tentative conclusion after reading this that I would recommend people place read-only data (like the initial snapshot of a VM) on SSD and then place everything else on mechanical drive, which would make managing storage on the SSD a lot simpler and should achieve really good performance, since there are no random writes. Obviously this takes a fair amount of focus/diligence to pull off though. Hope this is helpful, although I acknowledge it’s all a bit vague.
Great article. Can’t believe I haven’t read this before. Do you know much about the HP laptop selection? The only comparable to the Precision line I can find with 4 RAM slots is the Elitebook 8770w which only comes in 17inch. I like the idea of the 15 inch choice in Lenovo or Dell, but would like to go with HP.
I think there was an HP that I was looking at back then. 4500 rings a bell. IIRC we chose the Lenovo because it was ~£300 less expensive. I’ll see if I can unearth some of the original material from back then.
Hi again Tom. I was thinking of the Precision M4500, but as it happens, it didn’t have 4 SODIMM slots at the time, so we decided not to go with it. Back then we also looked at the EliteBook 8570w as well http://www8.hp.com/us/en/products/laptops/product-detail.html?oid=5257502#!tab=specs. It did have the four slots but I believe it was coming in significantly pricier than the Lenovo W520. We now have some W530 machines as well, but there have been some funky driver and stability issues with some ours. I’d be a little cautious about advising them now.
Ultimately, keep an eye on the processor. That’s what will have an impact on the number of memory slots. Hope this helps. We may be shopping around again soon. If so, I may put a new post up about the outcomes.