Robert McLaws: Windows Edition

Blogging about Windows since before Vista became a bad word

I Think the Harris Poll on Vista is Flawed

Mary Jo posted details on the new Harris poll about Vista awareness. Of 2,223 people asked, 1,934 (87%) were aware that Vista exists, which is a huge win for Microsoft. But it stops there: only 12% said they would definitely upgrade, down 8% from December.

I have a couple problems with this poll, however, which would lead me to question the conclusions that you'd be able to draw from it. For one thing, half as many people were asked in December as were asked in March. I'm not sure what that does for the statistics... maybe nothing. But it's not apples-to-apples.

The second problem I think is far more fundamental. I don't know that you can make an accurate comparison between the two sets of numbers on awareness and upgrade potential, because they didn't ask the same people 3 months later about whether or not they would upgrade. THAT would show a more accurate picture, IMO, because you would be able to draw more accurate conclusions about the effect of Vista's PR on the buying public.

Thirdly, I think the poll is flawed because it didn't break down the different responses based on the operating system the respondent was already using. Did 97% of Windows 98 users say they were going to upgrade? It also doesn't break down the percentages of operating systems use in December 2006. You might have been able to draw better conclusions from those changes too: For example, it was still possible to buy computers with XP in December. How many older OS users decided not to wait for Vista? How many had the option to Express Upgrade?

Finally, it include any statistics on the 3% of people who had already upgraded to Vista, including what drove them to upgrade, how they did it, what OS they were running before, etc. 3% of the internet-using population may not sound like a big deal to you... but 60 days after launch, that number is pretty significant.

All in all, I think the poll is interesting, but far too incomplete to draw a whole lot of accurate conclusions from. And that's surprising from an institution whose sole purpose in life is to gather complete data to draw conclusions from.

PostTypeIcon
3,264 Views

Comments

  • Wesley Shephard said:

    2,223 respondents is actually quite a reasonable number for such a poll. As long as the *methods* for obtaining the samples were the same, I would find it difficult to say this wasn't *apples to apples*.

    Check here: http://www.surveysystem.com/sscalc.htm, leave the population size blank (the number of windows users is huge, so it is irrelevant) switch to 99% confidence level and put in a confidence interval of 3 (meaning you want +/-3% on your figures). You will find that 1849 people surveyed would give this *very high quality* result.

    Since the same company did both surveys, it is probable that the same sampling methods were used, so I would put *much* faith in the numbers.

    April 4, 2007 11:47 PM
  • That's not the point. 1300 respondents took the December one, 2,223 respondents took the March one. More people took the March one by almost a 2-1 ratio.

    Besides, that's only one of my points that I argued. What about all the other ones?

    April 5, 2007 1:02 AM
  • Colin Theys said:

    The questions about current operating system and those who have already upgraded are good points, but the survey methods criticisms are largely unfounded. Wesley is right that both sample sizes should yield very high quality results and the specific number of people surveyed becomes irrelevant assuming that they were pulled from the same random pool. As for the returning to the same individuals previously surveyed, that is a different type of survey and one that is generally conducted on small groups of people to look at nuances of specific changes. If you're dealing with large groups of people, you will theoretically get just as accurate a result pulling new subjects randomly from the same pool as you will returning to the same group. Returning to the same subjects would simply add great cost and practically speaking would probably add error as some percentage would not be willing to participate again and / could not be reached you'd have to worry that the change group was non-random. By choosing a new random sample from the same pool they are following good survey methods. :) So I'm afraid I would have to say that it looks like a perfectly reasonably survey to me. That said, however, keep in mind that a difference of 8 +/- 6 is not particularly huge.

    -Colin

    April 5, 2007 2:55 AM
  • Wesley Shephard said:

    Statistically, the 2-1 ratio is irrelevant. Looking at that same confidence calculator I linked to earlier, punch in 95% confidence level (a 1 in 20 chance of the result being "out of range"), 1300 respondants and leave the population alone. Enter a percentage of 12 and you will find that the actual confidence interval is +/-1.77. That is the result of the *smaller* sample: the larger sample gives +/-1.35.

    Now, a poorly designed survey methodology could ruin these calculations, but the Harris people may make stupid *conclusions* at times, but I can rarely find fault in their methodology.

    An example of poor methodology would be doing the first survey in computer stores and the second via cold calls to homes during the day. Clearly the first sample would be more likely to upgrade: you added a selection bias for computer/home entertainment enthusiasts while the second adds a selection bias for the unemployed.

    A company like Harris is unlikely to make such a boneheaded error. With such large samples (and assuming proper methodology), the number of 98 users is going to be very similar (as a proportion) in both groups.

    Perhaps more interesting is the fact that 3% of the respondents *already have* Vista. From what I have been reading, this is consistent with the replacement rate for new computer purchases. In other words... the number of *upgrades* is in the doldrums, but the number of new computer purchases has remained consistent and thus the number of Vista machines (and users) will grow during the year as that replacement cycle continues it's normal course.

    Another thing to consider is that some people have been "burned" buying the "Basic" Vista-Ready configuration. Let's read Dells website regarding that configruation:

    Basic Windows Vista Experience - No Aero "Good"

    Memory: 512MB

    Graphics: Integrated or discrete

    Processor: >800Mhz

    Hard Drive: >15GB free

    Aero: None

    Great for: (And I quote here...) " Booting the Operating System, without running applications or games".

    Um, booting the operating systems *without* running applications or games? Who, praytell, is this marketed to? Worse, I have had clients and family *buy* this configuration and then have the *nerve* to be annoyed with the performance compared to XP and the lack of the features promised on the box.

    Let me repeat this: vendors are selling machines barely able to *boot* Vista to consumers. From the commentary I have received *this* is a large reason for the drop in upgrades. Many of those who did this to themselves are blaming Microsoft for the fact that their "new" machine is "broken" (their words, not mine). Some have salvaged XP disks and downgraded, but in others they are simply returning them (and even swallowing 15% restocking fees to do so).

    With this kind of "buzz" about what upgrading or buying a new machine will do happening with the hapless computer users, I'm not surprised that a bit of the excitement has gone away.

    April 5, 2007 11:10 AM
  • Wesley Shephard said:

    (I should point out that those who upgraded to the "basic" configuration... they didn't talk to me first.)

    April 5, 2007 11:13 AM
  • Colin Theys said:

    Wesley,

        I think you're dead on with that analysis and it is one of the things I think Microsoft really flubbed badly for the Vista release. One of the reasons people think macs are better is that when you get a mac, you know it'll 'just work.' ie apple doesn't sell computers that don't run their os well. duh. Microsoft should have done a much clearer job with the vista compatability stickers, what they mean, and what they will be granted to. They made the same mistake with XP. I work sometimes at a non-profit and they have a whole slew of absolute bottom-rung XP machines. They're HORRIBLE and can barely run the OS. They were sold with 128 megs of ram. Yes. 128. No surprise they suck.

    People see the sticker and buy it. But the sticker should not be granted to computers that 'boot' the os. They'll probably sell more computers immediately this way, but ultimately do serious damage to their image IMHO.

    April 5, 2007 10:32 PM
  • April 6, 2007 1:24 AM
  • Mihai said:

    "One of the reasons people think macs are better is that when you get a mac, you know it'll 'just work.' ie apple doesn't sell computers that don't run their os well. duh."

    Then you should see the dual processor G5 Apple was selling 2 years ago with 256 MB of RAM.

    April 6, 2007 12:20 PM
  • Mr Snarky said:

    Bobby, you're such a fanboy. When Ballmer farts, do your cheeks puff up like a chipmunk?

    April 6, 2007 6:28 PM