Stability

The purpose of the Stability section is to track bugs which are representative of our worst crashes. Every day we get thousands of reports from Firefox users when they crash. It's important that we triage and escalate the crashes that are having the greatest negative impact on our users. QA's has a shared responsibility to report, investigate, and escalate these top crashes as they become known. QA has a secondary responsibility to verify the crash is fixed when it's marked resolved. Sometimes this can be done manually if the bug is reproducible, or via crash stats when not.

Roles and Responsibilities

QA Lead

• Create and maintain Stability section in the Firefox test plan
• Attend meetings to discuss and escalate bugs of utmost importance
• Conduct daily triage of the explosiveness and topcrash reports to ensure critical crashes are being tracked
• Conduct daily triage of the topcrash and crash bug lists to ensure bugs remain on track
• Escalate critical or lagging issues to Engineers and/or Release Management as appropriate
• Ensure bugs are reproduced and/or verified fixed in a timely manner

Testers

• Assist with testing of crash bugs assigned to you
• Debug crash reports to find potential steps to reproduce
• Find regression windows when a bug is reproducible
• Verify fixed bugs by testing when reproducible or via Socorro when not
• Escalate bugs to the QA Lead when efforts have failed to reproduce actionable results

Meetings

• Crashkill - Mondays at 10:00 Pacific (excluding holidays) in Stability (via Vidyo) [IRC: #crashkill]
• Release Coordination - Tuesdays/Thursdays @ 10:00 Pacific in Release Coordination (via Vidyo) [IRC: #planning]

Daily Checks

Be sure you familiarize yourself with our topcrash criteria and ask for help in the #crashkill IRC channel if you are unsure about anything.

1. Review the explosiveness report and look for signatures which are either explosive (indicated in red) or new (indicated by 0.000 on previous dates)

2. Review the topcrash report and look for signatures which are either in the top-20 or rising/falling significantly.

3. When looking at each of the reports above:

• Report a bug if one does not already exist
• If a bug report does exist, update the bug with the latest data from Socorro
• If QA hasn't already tried to reproduce the crash, add the qawanted keyword to flag for testing
• If QA has exhausted all efforts to reproduce, flag an engineer to see if they have advice based on the stacks

4. Once done with the reports, review the list of topcrash/crash bugs in your test plan

• Bugs which have been marked fixed for several days should be verified fixed either by reviewing Socorro or by testing (if reproducible)
• Bugs which have been marked affected for several days should be updated with the latest data from Socorro and either flagged for QA or Engineering support as required

Test Plan Template

Topcrash Bugs (live example):

<bugzilla>
{
 "include_fields":"id,summary,cf_status_firefoxNN",
 "keywords":"topcrash",
 "f1":"cf_status_firefoxNN",
 "o1":"nowordssubstr",
 "v1":"--- verified",
 "f2":"op_sys",
 "o2":"nowordssubstr",
 "v2":"Android Gonk"
}
</bugzilla>

Crash Bugs (live example):

<bugzilla>
{
 "include_fields":"id,summary,cf_status_firefoxNN",
 "keywords":"crash",
 "f1":"cf_status_firefoxNN",
 "o1":"nowordssubstr",
 "v1":"---",
 "f2":"op_sys",
 "o2":"nowordssubstr",
 "v2":"Android Gonk"
}
</bugzilla>

Filing a Crash Bug

• To file a bug, open one of the reports and click one of the links under the Bugzilla section
• If you aren't sure which component to file the bug under
  • Signatures starting with JS will usually be in Core :: Javascript Engine
  • Signatures with GFX will usually go in Core :: Graphics
  • Core :: General is usually a safe bet, or just ask someone in #crashkill for assistance
• Be sure to include the stack information in the report (this can be copied from one of the crash reports)
• Be sure to include a link to "more reports" (this can be copied from one of the crash reports)
• Try to include any useful correlations, URLs, comments, or any other information you can think of that may aide in the investigation
• If the crash seems to be highly reproducible, set the QAWANTED keyword so we can find steps to reproduce and a regression window
• If the crash showed up recently, try to provide a pushlog between the last good and first bad build
• Important: Be sure every topcrash bug has a topcrash-* keyword, a status-firefoxN:affected flag, and is nominated to track