Releases/Firefox 2.0.0.12/Post-Mortem

This page is for notes from the Firefox 2.0.0.12 post mortem meeting held on Monday, February 11, 2008 at 3pm PST.

Agenda

General questions to think about:

• What did we do right? What worked?
  • QA Verification: A large bulk of security bugs got verified by QA. This is in contrast to some previous releases where 50% or less got verified. Maintaining this level is a Good Thing.
  • Automation: The fixes that went into automation helped it to succeed well this release for all our respins.
  • Release Timing: QA appreciated having bits on the mirrors earlier in the day so they didn't have to stay late to push.
• What did we do wrong? What didn't work? (Besides specifics below.)
  • Checkins (ongoing): A few major checkins went in at the last moment (day or three) before release which caused at least one re-spin. If these checkins landed earlier, it'd help catch regressions.
  • Showstopper detection: Improve the / Create a process to determine if a regression is a showstopper so there's very little work that gets done on an RC that might not be final. Formal 'stop' should be given as soon as possible.
    • Sam will work on a process.
  • Earlier/Staged Release: It'd be better on all to release in the morning. This would requires pushing the website and all bits to the mirrors early on and simply allowing users to force their update by using "Check for Updates" but not finish releasing (i.e., updating via daily AUS ping) until after 3pm. This is approved by IT and the client side already supports it. Need to determine what other work would be required.

More specific questions:

1. Did the new betatesters alias help?
   • Quick answer is "yes". At least one regression was found due to that list.
2. QA currently tests both Mac OS X 10.4 and 10.5. Is this overkill? Other improvements to the QA process? This will be covered in a QA meeting to be scheduled this week.
   • Al is heading the meeting this week.
3. We got stuck at product-details updates. How can we make that smoother?
   • Alert the web team a day or so in advance of when we plan to push so they can have someone ready to make these changes (clouserw or morgamic).
   • Sam will add this to the checklist.
4. How can we avoid publishing things prematurely?
   • Improvements to Kubla will help here. The first is allowing easier selection so the pusher doesn't feel rushed to get us to release and check everything. The second would be "grouping" checkins together so the pusher can easily see which items he/she *knows* can be pushed.
   • Sam will file a bug on grouping by checkin in Kubla.
5. How do we make sure all pages are valid XHTML before pushing them live?
   • We should strive for this but not block a release on it. Follow-up checkins can be made to fix XHTML validation issues.
   • Sam will file a non-blocking, low priority bug for incorporating this check into Kubla.
6. How can we make sure the l10n pages get staged in time for testing instead of being a rush?
   • We need to land the en-US relnotes (on the trunk) as soon as possible so that l10n pages can get done and staged as well.
   • Sam will update the checklist to better sort through this.
7. How can we better pick which copy of the whatsnew page should be used for a release (security vs. stability)? This was already answered on an email thread and needs to be incorporated into the checklist.
   • And the answer was... Alert jslater ahead of time to review the en-US page changes and make sure we haven't missed anything. Added to checklist
8. Are we using release-drivers too little now?
   • We should be careful not to go too far the "other" way and use release-drivers too little. Traffic should be "low" but useful and if it needs to be higher because there's more usefulness to it, we should up the traffic.
   • Sam will go through the release checklist and see which emails need to go where and note it in the checklist better.