User:Davel/PNSQC2006Paper
Contents
Making the Most of Community Feedback
Authors
- Dave Liebreich, Mozilla Corporation
- Various Authors, Mozilla Project
Abstract
The Mozilla open-source software project, which includes the Firefox browser, has a long history of quality-related feedback from its community. From its debut in 1998 when Netscape open-sourced their browser code to the world, many thousands of volunteers ready and willing to contribute to the future of the web have provided direct feedback to the mozilla.org development community. Many millions of users have downloaded and run the mozilla.org applications, and through such use have provided feedback as well.
This feedback is a competitive advantage for the project, providing insight into quality-related issues that would be difficult to obtain with a small, dedicated team. But the sheer volume of the feedback can make it difficult to process, identify, and prioritize specific issues to address.
In this paper, the authors present background information about the mozilla.org project, descriptions of the various feedback channels and tools, and walkthroughs of the processes and guidelines project members use to convert this feedback into prioritized tasks.
Bio
Dave Liebreich has almost 20 years experience supporting the development of complex applications and distributed networked systems. He is currently a Test Architect at the Mozilla Corporation, working on improving and extending the testing for mozilla project products such as Firefox and Thunderbird.
Copyright
- Copyright 2006 Mozilla Corporation
- First published in the 2006 Proceedings of the Pacific Northwest Software Quality Conference
Background
The Mozilla open-source software project includes the Firefox web browser, the Thunderbird email client, the Bugzilla bug tracking system, and many other web-based applications and technologies. Firefox, the most popular application in the project, is used by many millions of people throughout the world. Hundreds of thousands of people download Firefox beta releases, while tens of thousands download alpha and other early-milestone releases. Thousands download and test the nightly builds, and hundreds submit code to fix defects and add new features.
Most of the developers and testers are volunteers, devoting a portion of their free time to working on the project. Many large companies, including Novell, Sun, IBM, Oracle, and Google, employ individuals to work full-time on the mozilla.org codebase. Smaller companies whose products incorporate elements of the mozilla.org codebase also contribute feedback, testing, and code.
Feedback from these user, test, and development communities has been crucial in improving the quality of the applications. Translating that feedback into Bugzilla entries, the form used by project members to track work on the code, has been an ongoing challenge.
Feedback Channels
Bugzilla
One of the project norms is that developer work is coordinated in Bugzilla, a web-based defect and work tracking tool.
Early in the project, the development community chose to coordinate all work through Bugzilla. Bugs were filed for all changes to the code, including new features, changes to the build system, changes to the mozilla.org website, and fixes for defects.
Our instance of Bugzilla is almost completely open. A Bugzilla account is not needed to view bugs, so anyone who has a bug number can go look at the bug. They can read through the comments, look at the proposed patches to the code, and see exactly who is working on the issue.
Anyone can create a Bugzilla account for themselves, and the default permissions for new accounts include the ability file new bugs, comment on existing bugs, and add themselves to the list of people notified by email whenever a bug changes.
Community members -- mostly technically experienced software developers themselves -- were encouraged to record their feedback in bugzilla. In order to help prevent duplicate and invalid bugs from being filed, a Netscape QA Engineer (Eli Goldberg) wrote up bug filing guidelines (http://www.mozilla.org/quality/help/). These guidelines also help others triage the bugs so developers can better prioritize their work. This document has been maintained by Gervase Markham after Eli left, and links to it have been spread across newsgroup postings and the various mozilla.org websites. Most people who file bugs in Bugzilla have read the guidelines at least once.
Still, Bugzilla remains hard to use. Its complex input form and search mechanisms are discouraging to the less technical members of the community, as well as the more technical members who are in a hurry.
In order to make it easier to obtain feedback, other feedback channels developed and evolved.
With the advent of other feedback channels, the project developed mechanisms and processes to convert community feedback into Bugzilla reports. When bugs are filed based on information from other feedback channels, the source of that information is recorded in the Bugzilla report comments.
Talkback
Talkback, the crash reporting agent distributed with Firefox and Thunderbird, produces the largest number of feedback reports. Talkback was developed by a company named SupportSoft, and the Mozilla Corporation has a license agreement with them to distribute the agent with our products and run a server to collect incoming reports.
The Talkback client is included in official Mozilla builds, which include nightly builds, milestone builds, and release builds on the main development trunk and on the active release branches.
All Mac and Linux users, and 10-20% of Windows users, have Talkback enabled by default. Much like the Windows crash report dialog, Talkback fires up and provides the user with the option of sending in the crash log and adding comments describing what they were doing at the time of the crash. If the user agrees to submit the report, the data is sent to the Talkback server (talkback.mozilla.org).
Reporter
Not all quality issues are crash reports. Often, something just does not work right when viewing or interacting with specific web sites. In order to provide an easy and accurate mechanism to report problems while browsing, Firefox includes a built-in �Report Broken Web Site...”�tool.
If a user encounters a problem, he or she can select the Reporter tool from the help menu, answer a few simple questions, and the tool will send the information to the Reporter server.
Discussion Forums
Members of the Mozilla community do a lot of on-line communicating. There are many different channels to support that communication, each of which has its own monitoring mechanisms that guide quality-related feedback to the appropriate system for reporting that information
Web Forums
The MozillaZine Forums are probably the most accessible discussion channels for Mozilla-related topics. Community members often direct new users to these forums for support requests.
The forums are also used by programmers building applications that use the code in the Mozilla codebase, Mozilla developers, and community testers. These more dedicated community members read through the postings and relay the more significant bits on to other areas, including formal bug reporting in Bugzilla.
Newsgroups and Mailing Lists
Mozilla also hosts many newsgroups, all of which have an associated mailing list for participants who prefer email to NNTP (Network News Transfer Protocol) news.
These lists and groups are used to host most of the discussion of the technical issues involved in the Mozilla project.
Many times, a discussion of how to accomplish some technical task reveals a quality issue in the codebase, and one of the discussion participants files a bug.
IRC
A great deal of conversation takes place in IRC (Internet Relay Chat) channels hosted on irc.mozilla.org.
Hendrix
Some users would rather not open another application or log in to the MozillaZine Forums to provide feedback. For them, mozilla.org has set up the Hendrix tool. This tool takes information entered in a simple webpage form, obfuscates the submitter's email address, and posts the information to a public newsgroup.
By scanning through the structured newsgroup postings, mozilla.org community members can more easily identify issues from this otherwise low quality feedback.
Hendrix-based forms are embedded in the default home page for Firefox nightly and candidate builds.
Processing the Feedback
Bugzilla Triage
Hundreds of new Bugzilla bug reports are created each day. Many of them are invalid, or duplicates of existing reports. Some do not have enough information to be of help. Some are work tracking bugs entered by developers. And some are reports of problems that turn out to be high-priority issues.
Developers and testers use the Bugzilla system of products, components, keywords, and tags to search for, identify, and track reports of interest. Very rarely, however, are reports entered with the appropriate settings selected.
A small group of community members performs bug triage, which involves searching for duplicate reports, marking invalid bugs as such, and classifying the reports using the Bugzilla settings.
One interesting side-effect of an open bug tracking system is that the original bug reporter can view the discussions about the bug and monitor the progress of the effort to resolve it. The original reporter can even see the email address of the developer working on the fix.
Triaging crash reports
The mozilla.org Talkback server receives a lot of crash reports, and the mozilla.org community has developed guidelines and approaches for triaging the incoming information to get the most return on time spent analyzing crashes.
Talkback triage starts at the main Talkback report site. Developers and testers check the list of most frequently reported trunk build crashes around 3 times each week, and branch crashes as often as daily during the release process. The product release teams also use the crash frequency reports in the nomination and approval process for maintenance release fixes.
The people who check the Talkback reports file bug reports in Bugzilla for crashes that contain complete stack signatures, even if there is not enough information to reproduce the crash. Developers and interested community members watch for such bug reports, and investigate the code implicated by the stack signature and other bug reports that might share the same root cause.
For Talkback reports with incomplete or corrupt stack signatures, the mozilla.org community relies on the extra information provided by the end user. A community member usually will include the end-user comments when filing a bug report based on a talkback incident �from the field”, and community members regularly include talkback incident IDs in their bug reports.
Release teams also pay attention to changes in the topcrash lists from release to release. If a topcrash without an explicit fix drops from the list in a new release, members of the team try to figure out which fix might have addressed that crash�s root cause. And new crashes in a new milestone build are given extra attention.
Talkback provides huge value in debugging and isolating stability issues, and it has been used to identify and debug thousands of major crash bugs in Mozilla over the years.
Reporter Triage
The reports on the Reporter server are often used when investigating a problem report from a single user in a different forum. If the site in question is in the �Top 25”�list, then the problem is probably not specific to that one user.
Reporter data is also useful when contacting a web site to ask them to change their content so that it will work with the mozilla.org applications. Such evangelism requests tend not to be ignored when they are backed up with a large number of unique reporters of problems.
Discussion Forum Triage
Many experienced community members �hang out”�in the various discussion forums. Some actively answer basic questions and help new participants navigate through the wealth of information available about the Mozilla Project.
These people, and other community members who just scan the forums, also take ownership of certain issues, and either file Bugzilla reports, or bring up the issue in a more appropriate (and possibly more technical) forum.
People Providing Feedback
There are long-term mozilla.org community members who have gravitated towards specific product functional areas or product activities.
For example, there are folks who work on layout and rendering issues, finding new bugs and reducing existing test cases. Some other folks work on security, and others work on the build system, and so on.
There are also groups of people who work on testing the nightly builds, looking to be the first ones to report problems.
The Mozilla QA group has organized test days and bugfests to guide new community members in providing feedback.
All of these groups of people make extensive use of the feedback channels listed in this paper.
Nightly Builds
The nightly builds of the mozilla.org products are available to the public. When a patch is committed to the codebase, the original reporter of the associated bug is often asked to check if the fix works for them in the next build.
Summary
The feedback from millions of users is an incredibly valuable asset to the Mozilla Project, and one that requires a significant amount of effort to use effectively. By setting up tools to help users provide feedback, encouraging community members to form a hierarchy of people performing feedback triage, and monitoring feedback reporting sites, the project has been able to convert this feedback into a huge competitive advantage.
Acknowledgements
The following people contributed content and/or provided feedback on this paper:
- Asa Dotzler
- Gervase Markham
- Jay Patel
- Eric Shepherd
Appendices
Mozilla Project "Legal Entities”
The Mozilla Foundation is a non-profit corporation that provides organizational, legal, and financial support for the Mozilla open-source software project.
The Mozilla Corporation is a wholly owned subsidiary of the Mozilla Foundation. This corporation employs a small number of people (~50) who dedicate their time to the project. Most of these employees are long-time community members.
Various non-profit organizations exist to promote, develop, and help deploy Mozilla products in specific regions.
