Hmm, now it's Vista winning?

As anyone who’s ever worked in a PC performance lab knows, the #1 rule of benchmarking is: Your mileage may vary.

I remembered that rule when I read my colleague Adrian Kingsley-Hughes’ epic account of his benchmark tests of Windows Vista SP1 versus Windows XP SP2 (Part 1 and Part 2). The first thing that struck me was how far apart his numbers were from those I was getting. In fact, I went back and redid all my tests to confirm that I hadn’t missed anything along the way. They checked out completely. On my test bed, with only one exception, Vista SP1 was consistently as fast as or faster than XP SP2, a result markedly at odds with Adrian’s findings.

When I re-ran the tests after making sure that these tasks were no longer running in the background, I got the consistent results shown here.

I agreed completely with a remark Adrian tossed off almost as an aside in the middle of Part 2 of his report:

[O]ddly enough, Vista SP1 felt more responsive [than XP SP2] to user inputs such as opening applications and saving files while the tasks were being performed (we tried this out on separate runs). Problem is that it’s darn hard to measure this end responsiveness without relying more on synthetic benchmarks.