How not to do a performance test

Written by Carl Campos

Monday, 19 May 2008 11:55

Via 4Sysops, I read an article written by InfoWorld's Randall Kennedy, where he calls Vista "a bloated pig of an operating system." Without a doubt, Vista runs slower on the same hardware than XP. However, when you use the phrase "bloated pig," you'd better be sure you've got your performance tests in order. Unfortunately, I found so many mistakes with his testing methodology that Mr. Kennedy's results are meaningless.

I put the same disclaimer in every article I write about Windows Vista, but I don't want to be seen as a Vista cheerleader. I'm not advocating for a political cause here or fighting some pointless computer software holy war. I just want to see people do the RIGHT THING. Much of what I see in Vista "journalism" are a set of pre-determined conclusions (Vista sucks!) with evidence added in later to support that conclusion. To be perfectly clear, I think Mac OS X is generally better for home users and Windows XP's performance is better. But Vista does more, looks better and is more secure than Windows XP, which is why I use it.

I'm no performance testing expert, but I think Mr. Kennedy's methodology is completely wrong. You can find his published test results here. When you're testing whether the *operating system* is faster, the only variable should be the *operating system.*

Kennedy tests Windows Vista with Office 2007, but Windows XP with Office 2003. How does he know the "bloated pig" performance problems are with Windows and not the new version of Office? Based on the results he's published, there is no way to differentiate between Office performance problems and operating system performance problems.
The author tests Vista and XP on virtual machines, meaning that his results are completely dependent on virtualization/emulation drivers provided by the virtual machine vendor. This assumes that the drivers for virtualizing Vista hardware are as good as those virtualizing XP hardware, which there is no way to know. When you performance test XP versus Vista on a VM, you're measuring virtualization performance more than you are operating system performance.
The author tests with a single core (virtual) CPU. You can buy dual core PCs for less than $400. In fact, unless you buy something like an Eee PC, it's difficult to even find a new PC with a single core CPU. You can make an argument that his tests apply to upgraders, but if that's all you're testing, you can't really reach the conclusion Vista is a "bloated pig." Why not test Vista versus XP on the hardware that people can go to the store to buy today?
Along the same lines, Kennedy tests Vista with 1 GB of RAM, which only makes sense as a data point, not as the data point. It's well known that Vista runs better on 2 GB of RAM. You can argue the merits of having to spend another $40 on RAM, but again - it makes sense to test Vista on the hardware people will actually use. Looking at the author's own results - see how much better Windows 2000 runs than XP SP2 in 256 MB of RAM? Ignoring the Office suite difference (for now), in three of his four tests, Windows XP SP2 takes 50% longer to accomplish the same task as Windows 2000 with 256 MB of RAM. It isn't because Windows XP SP2 is a "bloated pig of an operating system." It's because XP doesn't run particularly well in 256 MB of RAM, and Vista doesn't particularly run well in 1 GB of RAM. Test them both with 2 GB of RAM.

In summary, here's how I'd setup a fair, apples to apples XP vs. Vista comparison: Test by installing both operating systems directly on identical hardware, not in a VM. Test 3-4 year old hardware that was released around the time of Windows XP SP2 and modern hardware (dual core, 2 GB of RAM) that people would buy today. Test both Office 2003 and Office 2007 on Vista and XP. Test with 512 MB, 1 GB and 2 GB of RAM. As I said, I'm sure XP will still come out ahead on modern hardware, but not by nearly as much. I don't get paid to test hardware for a living, but it's pretty easy for me to see that you can't draw any reasonable conclusions from the methodology Kennedy used, let alone his "bloated pig" quote. If you're going to jump on the "Vista sucks" bandwagon, at least have the numbers to back it up.