CrashKill/2012-09-04
Welcome to the first Stability Newsletter! For some time now, the crashkill team has been thinking about publishing a regular newsletter to the broader Mozilla community. The goal is to engage users and developers in helping us improve the stability across all our products.
- The newsletter will cover a wide range of topics relating to crashes and crash analysis.
- We might focus on a particular problem, a specific product, discuss crash trends or simply provide a list of bugs we want help reproducing. The newsletter will be published twice during a full release cycle.
- The first one will go out on the Friday a week after we merge. This enables us to report on top problems once we have enough users on the first build for that channel.
- The second newsletter will go out later on during the cycle to highlight crashes or crash spikes on Beta. This might be a call to action for help reproducing tough issues.
Contents
Stability 101
Since this is our first newsletter, we thought it would be good to cover a few basics for those who may not be as familiar with crashkill and our activities.
- Meetings and Communications
- The crashkill team meets weekly on Monday morning at 10:00am PT. Those in MV meet in Zombocom-3Z and remote folks dial into the Zombocom vidyo room. (NOTE:Setup crashkill vidyo room)
- Organized crash triage is on an ad-hoc and as needed basis. Daily crash chatter happens in IRC. See IRC channel info below.
- Members of the crashkill team attend and report on stability during the twice weekly Channel Meeting (Tuesday/Thursday @ 2:00pm PT) as well as the weekly Firefox Platform Meeting (Tuesday @ 11:00am PT).
- Queries and Tools
- We use Socorro as our primary tool to run reports, search and track stability issues.
- We often have the need to generate a number of manual reports to help analyze and diagnose a particular issue. Kairo has many in his own directory.
- Manual reports are often temporary until we can integrate them into Socorro. Other times we need a manual report to track a short-term issue.
- We used a number of keywords in bugzilla to track crashes - crash, topcrash, regression, reproducible and hang.
- We also used a wide range of whiteboard tags to help us inventory crashes. These are often less consistent. Some like [startupcrash] are more recent and some like [crashkill] are old and not used that much any more. We will often use temporary tags to track crashes that correlate to a particular OS release, [Win8] for example.
- Reporting Crashes
- If you crash, please select to send the report. Adding comments is also very helpful.
- If you log a bug for a crash...
- Include the format - crash in "top signature stack" - in the bug title. It will look something like - crash in JSContext::saveFrameChain.
- Please include the crash keyword in the keyword field
- We have a crash signature field which links bugs to signatures in Socorro. The format is a list of signatures like [@ signature], ie: [@ JSContext::saveFrameChain]. There might be multiple different signatures that associate back to the same bug.
- Wikis
- The crashkill wiki is a pointer to meeting notes, queries, reports and other analysis.
- IRC
- We hang out in #crashkill on irc
- The Socorro folks hang out in #breakpad
Desktop
Flash
The increase in Flash related crashes with the release of Flash 11.3 continues to be a problem.
- We have been working closely Adobe to address these. Subsequent updates to 11.3 fixed a number of problems.
- We are tracking the crash rate closely - https://crash-analysis.mozilla.com/bsmedberg/flash-summary.html
- The adoption of 11.4 released last week is still relatively low so we haven't yet been able to measure the full impact.
- Hangs continue to be a major issue and we will be tackling this more aggressively.
- Our top Flash crashes are...(placeholder list of bugs)
- Out top Flash hangs are...(placeholder list of bugs)
Windows 8
- Talk about progress on Windows 8 stability issues
- How users can tag bugs
- What we need help testing...etc.
Tracking Aurora - Fx17
A week ago we merged the Fx17 train from m-c to Aurora. This is a list of topcrashes we are currently tracking for Fx17. We would use help isolating regression ranges or identifying reproducible steps.
- (List of top crashes we want help with)...
Tracking Beta - Fx16
Our first beta build is out with xxx users. Here are the top signatures we are tracking and could use help investigating and reproducing.
- (List of top crashes we want help with)...
Mobile
FirefoxOS
Some of you may have noticed a new Product listed in Socorro - B2G.
- As of today, we only support crashes generated from the Desktop builds and not device builds.
- There are no crashes logged yet but this shows we have some of the mechanics in place already.
- We are aiming to have device crash reporting up and running.
- ...getting more updates from team in Brazil