Robert McLaws: Windows Edition

Blogging about Windows since before Vista became a bad word

XP SP3: The Problem With Benchmarking Beta Software

A lot of people lately are making a big deal about a recent report that Windows XP SP3 has a greater performance improvement over XP SP2 than Vista SP1 has over Vista RTM. Now, the AP has gotten their hands on the story, providing the masses with a vapid and misleading summary of the "facts". Personally, I think focusing on it at this point is a really bad idea, for seven reasons:

1. With Windows, It's Beta Code, Even If It Says "Release Candidate"
Historically, Microsoft doesn't tighten the screws on Windows until VERY late in the game. In July of last year, I posted that I thought Microsoft needed to delay Vista and add another short beta to the cycle. It caused a lot of stir, and generated some great discussions. A month later, the Release Candidate builds changed my tune. The thing is, there is still a bunch of debug and instrumentation code in the current SP1 builds that will not be there in the final release. It was true with Vista then, and it's true with Vista now. So you can't necessarily judge performance based on that, because the final builds won't have all that extra instrumentation in there, which will lower overhead and boost performance.

Furthermore, with Windows, a Release Candidate doesn't really mean "we think we'll ship this particular build unless something bad happens," it just means "we're almost done, and this is close to what you can expect." There will still be changes between Vista SP1 RC1 and Vista SP1's release. So benchmarking right now is kind of pointless, and quite a bit irresponsible.

2. The First Service Pack is About Reliability, Not Performance
As I recall, XPSP1 didn't feature any terribly big performance gains, it was all about fixing the stuff they didn't have time to fix before they shipped. Vista SP1 is mostly the same way, but has the added benefit of receiving the improvements from the Windows Server team as well, hence the networking improvements. (Both Vista SP1 and Windows Server 2008 carry the same version numbers on their binaries).

3. XP SP2 was Longhorn Minus WinFX
Microsoft has a roadmap for Windows years before they actually know what's going to be in it. Originally, Longhorn was supposed to be a minor point release to Windows XP, bigger than a service pack but smaller than a major release. (XP SP2 should have really been XP R2, but that's beside the point). After the MSBlaster virus, Microsoft decided to pull a bunch of the stuff they had planned for Longhorn (Security Center, Firewall, DEP, etc) down into XP to get people safe on a shorter timeframe. So Microsoft had the opportunity to have Windows XP benefit from the work they were doing on Longhorn, which is one of the reasons why it is so stable now, and SP3 will make XP even more so.

4. XP SP3 Derives From lessons Learned With Vista
The latest Service Pack for XP obviously gets some of the benefits of the lessons learned with Vista. There are better compilers, better code metrics, better techniques, and so forth. XP also has fewer things that need to load into memory, and a smaller codebase. While those may sound like pluses, XP also doesn't have the Error Reporting, Problem and Solutions checker, or depth of Performance and Reliability monitoring of Vista, so they're really not. Microsoft can't use the error reports to improve XP quite like they can with Vista.

5. Windows XP Has Matured Already
Any operating system that has had 6 years of development is going to get more robust over time. This service pack is the end of the line for XP, so of course they're going to squeeze every last ounce out of it they can. If it's already as stable as it's going to get, what else are you going to focus on?

6. Vista SP1 Focuses On Performance Improvements I Care About
Personally, I could really give a rats ass if Vista SP1 makes Office 2007 start 30 microseconds faster. In my daily life, I have to move a lot of files around my network. I stream stuff from my Media Center, I download files off of MSDN, I buy TV shows from Amazon Unbox. I love the fact that Microsoft has improved network and SATA throughput in SP1. My disk-to-disk transfers on the same machine consistently hit 30-40MB a second, and transfers between machines hit 15-25MB/sec on my gigabit LAN. The fact that copying a Virtual Machine from one box to another went down from 90 minutes to 20 saves me a crapload of time, and therefore has a direct effect on my bottom line. That's a far more tangible benefit than my Office documents opening sooner.

7. 80& of the Human Population Still Does Not Have a Computer
Many people are freaking out by saying that XPSP3 is going to hurt Vista sales, and I think that anyone that thinks that needs to remove the blinders they have on. There are 6 BILLION + people on the planet, and a little over a billion computers out there. Microsoft is still a gi-normous company, and 80% of their market is un/under-served. Continuing to refine older products to benefit existing users is not going to hurt a company that has a LONG way to go before each person has a computer.

So I disagree with just about everyone that thinks it's a big deal. It's really not, and I hope this provides some context. And XP SP3 is not a huge mistake, it's just good business.



  • Fabio said:

    Right on the mark Robert.

    November 30, 2007 2:45 PM
  • Aaron said:

    This is a good article, but on the last point - of those 6B people, more than half will never own a computer let alone ever see one.

    November 30, 2007 6:26 PM
  • Bryan said:

    There are a few things I would love to see Microsoft tackle with Vista:

    - Fast User Switching: This is so incredibly slow under Vista when compared to XP's implementation, they really need to fix the speed here considerably.

    - Delay between running a setup program and the UAP dialog displaying could do with improvement, I'm often left wondering whether I did indeed ask it to run.  Poor user experience here.

    - Simplify the networking side of things.  This has been written to death, because it is true.  There is a need to go to too many different windows to achieve something, this should not need to be the case.  Build a Network Centre, one window, everything accessible there.

    - Laptop/Notebook info, can I get told when I am running via battery how long I have (est) left before battery is expended?  The whole % of battery remaining is only slightly useful.

    - Media Center: A great environment.  I hate the fact that I cannot select a channel from a list, instead I have to remember the number.  Here in Australia we don't have a huge number of channels to watch, but we've got multi-channel digital tv so the number is increasing, it's a pain to remember the number of the channel I want.  Also, it would be nice if there could be a 'quick record' functionality whereby I can hit record on the remote and it goes for 30 mins, a second time increases by another 30 mins, the need to always set start/stop times is annoying.  Oh, can Australian's also have a damned tv guide that works, it would be so useful!

    - IE: I'd kill for the whole Firefox live bookmarks feature in IE.  If I had that feature I'd probably have switched.

    - Search:  Would be nice to hit the windows key and type something and control panel applets are included in search results

    I dare to dream!

    November 30, 2007 6:52 PM
  • Aaron,

    Even if you only hit 3B, it took almost 40 years to reach the first billion, it'll take 10-15 to reach the second.

    November 30, 2007 8:42 PM
  • Mitchel Tyrell said:


    Get live bookmarks in IE7 with this:

    November 30, 2007 9:07 PM
  • Imran said:

    Right on !

    December 2, 2007 7:09 AM