Breakpad/Status Meetings/2008-Mar-19

Key topic - what it takes to make fx3 beta5 and final more stable.

 Meeting details:
* 2:00p PDT, 5:00p EDT 
* 650-903-0800 or 650-215-1282 x91 conf# 249 (US/Intl)
* 1-800-707-2533 (pin 369) conf# 249 (US)
* irc.mozilla.org #breakpad for backchannel

Progress on Top Crash Analysis -- Beta 4.

• Landing the comments in the summary reports last week was a big help in speeding up the analysis
• Sam's wiki page for summarizing status of the analysis work on each release is also working fairly well - See: http://wiki.mozilla.org/QA/Topcrashes
• This can serve as a work around until we get solutions for bugs 448156, 411357
• Many of the top crashes for Beta 4 have to do with Extension/Plugin compatibility, and corruption with installation setup. Extension and Plugin Blocking and a few key bug fixes that are in progress will should make beta 5 much more stable.

Crash Analysis Needs

• One additional thing we need is some semi-efficient way for getting mini-dumps to dbaron and others - https://bugzilla.mozilla.org/show_bug.cgi?id=411431

• The other thing that would help to spot key crashes is to add "time since start up." Right now thats broken and reports '0' for all crashes. Getting this in place would help us to spot start up crashes and understand more about why some crashes are getting no commments. https://bugzilla.mozilla.org/show_bug.cgi?id=419070

<bold>lars fixed this and it has been pushed - whoohoo!

• we also need to figure out how to get some comment and URL analysis ramped up to find common actions and sites that are resulting in a variety of stack signatures. https://bugzilla.mozilla.org/show_bug.cgi?id=423502 and https://bugzilla.mozilla.org/show_bug.cgi?id=415027

Release Comparison Analysis

• For overall release metrics that can help us compare releases we need to get some MTBF calcs and reporting in place. https://bugzilla.mozilla.org/show_bug.cgi?id=411424
  • Talked to Ken this week to provide some background and get him involved in creating some MTBF numbers and help in validating these "release metrics"
  • Morgamic is also thinking about the best way to do these non-dyamic query reports that give us a sapshot of overall release metrics every day or hour.

Sorocco Scalability

Morgamic has some concerns/questions/work going on ensure the system scales as we ramp from 600k daily users to 50-100M users. We need some metrics that give some insight into how things are scaling as the load starts to ramp. We should also discuss morgamic's idea for throttling based on update channel, or any other throttling mechanism we want to try and implement.

Quick Review of other Feature and Bug List and Priorities

[ https://bugzilla.mozilla.org/buglist.cgi?cmdtype=runnamed&namedcmd=socorro | Current Buglist ]

Action items

• Ted to look at feasability of keeping minidumps for topcrashes
• Ted to investigate aggregate URL reports
  • top N sites
  • alexa top100 sites - talk to jay about this
  • sites in >N reports
• chofmann to get kkovash to look at MTBF calculations, regression analysis
• chofmann filed https://bugzilla.mozilla.org/show_bug.cgi?id=412605#c6 to break out the idea of creating an analysis report to spot problematic .dll's in the process list
• ss to add Firefox 3.0b5 to query form as soon as we start getting beta5 builds.
• ss or chofmann - get blocklisting requests on file for the rest of crashing extensions and plugins in b4