Late Monday evening, I was writing bug reports for a private, Vista-related beta test that I'm involved in, when I noticed a new feature on Connect. Their last release enabled users to view feedback in a variety of different ways. So I started looking through the list of most recent bugs, and leaving comments on bugs I had also noticed. As I was paging through the list, I wondered what a statistical analysis on all the Windows Vista bugs in the system would look like. It wasn't the kind of question I would typically leave hanging out there... the more I thought about it, the more curious I got. I had to know.
So I embarked on a journey... one that would test the limits of my patience, Connect, and my dual-proc computer. I whipped out Excel 2007, and started to copy and paste all the bugs from Connect into a new spreadsheet. There were 100 bugs to a page, and 287 pages. You do the math. Oh wait, you don’t have to, I already did. More on that shortly.
<side rant>
So what started out as a good idea soon became a tedious affair. You see, as cool as this list was, it still didn't help the fact that Connect is a badly-written application. I'm sorry guys. I know there is a team that put their heart and soul into making it happen, and I know I rag on you all the time. But whoever built it doesn't know ASP.NET, and it shows. The paging system for the list is absolutely terrible. Only the "forward" and "back" buttons work; the "beginning" and "end" ones don't. There aren't clickable page numbers, so you can't jump ahead. And because it is all postback-based, you can't specify a query string value to jump you to a specific page. And, oh yeah, did I mention that the site makes you log back in every 60 minutes? Which means every hour you have to start back at page one. When you set the bug list to 100 items per page, the average page load time is 9 seconds. At that speed, you can go through 6 pages a minute. If you have to start at page 120, it takes 20 minutes to get back to where you left off.
</side rant>
Four and a half hours later, I had copied all the bugs into my spreadsheet. Then, after using the bug IDs to remove duplicates from pasting errors, and using the bug titles to remove other duplicates (Excel 2007 found 1,072 duplicate bugs in the system), I used PivotTables and PivotCharts to analyze the data. Here's what I found out:
|
Windows Vista Bugs (As Of 3 July 2006) |
|
Active |
5,743 |
|
Closed |
20,176 |
|
Resolved |
1,020 |
|
Total |
27,479 | |
|
(click to enlarge) |
I was really surprised that a little over 20,000 bugs have been closed so far. I really didn’t expect it to be so many. It’s also important to note that, with 4 months left to go in the development cycle, 1/5th of the total bugs submitted are still open. But, Microsoft is closing bugs pretty quickly, as only about 250 of those bugs are more than two months old. Finally, since I pulled this information two days ago, there have been over 150 additional bugs entered into the system, some of which have already been resolved or closed.
That data was interesting, but it wasn’t enough. I saw an awful lot of bugs posted last week, when the June CTP was released on Connect. I wanted to see if there was a correlation between bug submissions and new builds. So I created a PivotChart to plot out the daily bug stats. What I discovered was pretty cool. I’ve overlayed build release dates corroborated from Brandon LeBlanc’s Windows Vista Build Tracker over on The Hive. Note that all publically released builds are notated by their CTP name, TAP/interim builds are notated only by their build number.
(click to enlarge)
Sure enough, with the exception of leaked TAP build 5259, every single build release corresponded with a large increase in new bugs entered. 5259 was an interesting discrepancy, because initially, it spiked higher than 5270. But when I was copying the bugs into the spreadsheet, I noticed that a lot of the bugs around that time cycled through the same 3 titles. After removing the duplicates (there were almost 400 of them), 5259 became a negligible blip.
All those spikes might distort the picture a bit, but from this data, I calculated that testers submit 81 bugs a day on average. Several other conclusions can be drawn from the graph. Some interesting points:
-
With the exceptions of the gap between the December and February CTPs, as well as the lag after Beta 2, Microsoft has consistently dropped a new build about every 6 weeks.
-
Within 24 hours of a build’s release, at least 200 new bugs are added to the system. It appears that testers are quick to log issues, and then settle into a routine of using without reporting.
-
5270 saw the introduction of the Start Orb. It was also the first time a CTP went over 300 bugs in one day (353 the first day, 338 the second)
-
With the exception of June 3rd, the bug pace has taken a significant upturn since Beta 2, and the pace picked up even more when the CPP became available. This was to be expected, since Beta 2 was the first time Windows Vista was available to the general public.
That last point is an important one for several reasons. First off, the bulk of the open bugs have been entered into the system post-Beta 2. This means that Microsoft still has their work cut out for them. Second, though the average for the entire period is 81 bugs a day, a closer look at the data since May 1 shows that the number of bugs per day is actually increasing, not decreasing, trending closer to 200. While the amount of feedback is fantastic, and definitely expected based on the number of people with access to Beta 2, one would hope that, at this stage of the game, those number would start to go down, not up. This is illustrated in the graph below.
(click to enlarge)
Now, one might look at all this data and think that the builds are getting buggier. But I don’t think that is a correct assumption. I don’t think there are more bugs, just more people with access to the builds. And the Feedback Reporting Tool Microsoft quasi-packages with the Vista builds doesn’t let the user search for existing bugs before posting new ones, so it is hard to tell how many separate entries with different titles are posting about the same issues.
Even so, the picture is far from complete. The data raises new questions, ones not likely to be answered. For example:
-
Because Connect doesn’t show the close date in the summary pages, it is impossible to do a more meaningful analysis of the bugs, such as bug age or close rates. The important factors in determining the success of the program lie in seeing how long it takes to resolve issues, and how many bugs get closed each month.
-
Microsoft’s two-level closing status doesn’t allow for a higher-level analysis, like the number of duplicate bugs or the number of not reproducible issues. Further, it is apparent that Microsoft is not accurately differentiating between closed bugs and resolved bugs. If Closed = “Can’t Reproduce” and Resolved = “Fixed”, that means that only a little over 1,000 customer issues were actually addressed. That can’t be right.
-
Connect isn’t tied in with Microsoft’s internal database. There’s no telling how many bugs are in that system, either.
-
Microsoft has a ton of new built-in error reporting functionality that automates the process of sending in crash data. I know that personally, when I have a crash like that, I don’t usually file a bug because WER already caught it.
I’m sure Microsoft has more internal data on the subject (and if they don’t, I’d be more than happy to help, since I’m currently in the market for a job ), but I doubt they will be able to respond to any question like these publicly (although it would be damn cool if they did). Either way, Microsoft should build tester-accessible reporting tools into Connect, so that testers can get information like this without having to spend two straight days putting it together.
In closing, this analysis reveals much about customers are doing more than just kicking the tires on the latest Windows Vista builds, but not enough about how Microsoft responds to the reports that are filed. But I don’t need numbers to tell me the builds are getting better. As I’ll discuss later (if I’m not castrated by MS Legal for publishing this, of course) 5456 is a huge leap forward in terms of build quality. And if I can figure out how to leverage some of the more advanced PivotTable functionality in Excel 2007, I might post some more charts based on the data I’ve gathered.
This analysis does, however, make a strong case that Microsoft is clearly committed to finding a solid middle ground between the open- and closed-source models… though their vehicle by which they do so (Connect) leaves much to be desired. It proves that a software company can become more transparent and get customers more involved, while still maintaining control over the source code. The Windows team still has a *long* way to go in terms of transparency, but as I’ve shown, they are making measurable progress.
If only every team at Microsoft published their bug database…
UPDATE: Read Part Two: The Aftermath