The goal of this project is to engage with users who provided an e-mail address in the Crash Reporter by

• Providing relevant info about their crash as well as general info about how to avoid crashes in Firefox.
• Letting them know when their crash has been fixed in Firefox.

The project will require changes to SUMO and Socorro in order to supply the required functionality. For the secondary goal, Session Restore and/or about:crashes need to be changed, too.

Engage with users after Firefox crashes

We should engage with users via e-mail in the following situations:

• After submitting the crash report to provide info about the crash, including relevant support documentation links for the crash signature and generic support documentation for how to avoid crashes in general.
• When new support documentation is available for the particular crash signature (if it didn't exist when the report was submitted).
• After a bug has been fixed and shipped in a Firefox release, to inform the user that the bug is fixed in Firefox version x.y.z. This will also work as a powerful way of retaining users, including getting users back who may have switched to another browser between the crash report and the fix.

When submitting the crash report (Q1 2011)

This e-mail will be automated:

1. A user submits a crash reports and provides an e-mail address.
2. Socorro generates the signature for the crash and queries the SUMO search to see if there's an associated support article for this particular crash.
   • If an article exists, the title and URL of that article is fetched and used in the e-mail.
   • If an article does not exist, the report is flagged accordingly so it can revisit the report when new info is available.
3. An e-mail is sent out pointing the user to the support article (if one exists) and also includes general information about how to avoid crashes (linking to a generic article on SUMO listing ways to keep your Firefox, plug-ins and add-ons up to date), where to get more info about the crash itself (linking to the crash report on Socorro).

Generally, the e-mail should:

• Thank the user for submitting the report and possibly describe what's going to happen to set the right expectations.
• Inform that we will contact the user again if there's more info about the bug.
• Make it clear that it's not possible to reply to the e-mail and that they should go to support.mozilla.com for help.

Issues with regard to authenticity of email / phishing can be addressed by encouraging users to "visit mozilla.com and click on blah for more information" (taking a lead from Paypal/eBay here). We should make sure that emails come from a mozilla.com address and use whatever we have in place for email authentication.

When new support documentation is available (Q4 2010/Q1 2011)

This e-mail will be automated:

1. A cron job would check each signature that lacks an associated SUMO article to see if new info is available.
2. If a new article is detected, all crash reports with that signature and an e-mail address will be queued up for another e-mail message including the new information.

Generally, this e-mail should make the new information prominent, but otherwise retain the language and key messages of the initial e-mail.

When a bug has been fixed in a shipped Firefox release (Q4/October)

This would in part be a manual workflow:

1. When a release is planned generate a list of fixed bugs (from Bugzilla)
2. Provide facility in Socorro to search by Bugzilla bug for a list of associated crashes
3. For each associated crash, generate a list of user submitted emails (minus any users who have said they don't want to be emailed again) - this function should be available only from the admin interface
4. Mark those emails as contacted for that given crash, so we don't contact them again.

Technical requirements

Changes to Socorro

UI changes consist of three basic web pages:

• Web page to allow email setup. User would input:
  • Crash signature
  • Start and end dates
  • Email subject
  • Email body (text only for this iteration)
  • Hitting submit would take you to the preview page
• Preview page will show the user how many emails will be sent that match their criteria and have a confirm button that will submit the email for sending
• Previous mailings that have been sent, just a list in reverse date order. Users should able to click through to see the text that was sent.

Back end changes:

• New database tables:
  • Mailings
  • Users x mailings
  • Users unsub (user email) - users not to email
• Email injection code

Full details can be found at http://code.google.com/p/socorro/wiki/CrashFixedEmailDesign

Changes to SUMO

• Add a webservice call that, given a signature and locale, searches for relevant content on SUMO and returns a URI. The web service should work as follows:
  1. If a canonical article exists for crash_signature (top search result), return the title and URL
  2. Else if other search results exist for crash_signature (e.g. forum threads), return URL to search results
• Should be available sometime the week of March 14th.
  • See bug 631341.

API

Will be available as of SUMO's 2.6.2 release. Scheduled for 15 March 2011.

GET /postcrash?s=<signature>

Return values:

HTTP/1.0 200 OK
Content-type: text/plain;charset=utf8

http://support.mozilla.com/...

or

HTTP/1.0 404 Not Found

A request without an s parameter will result in

HTTP/1.0 400 Bad Request

Responses may set caching headers. Conditional GET is encouraged. Consumers should wait before requesting again in the event of a 5xx error.

Timeline

• Monday Sept 13: Meeting with QA to establish QA/Support workflow
• October: Socorro 1.7.4 freeze (Socorro)
• October: Socorro 1.7.4 push (Socorro)
• October 22: Finalize e-mail template to send out for fixed bugs (Michael/David)
• November 22: Finalize e-mail for [Russian crash] (David)
• November 22: Finalize support e-mail FAQ and support article for Russian crash (David)
• November 24: Have e-mail + support article + FAQ localized into Russian (Alexander)
• November 30: Test e-mail campaign to a select number of people to ensure that the e-mail encoding is correct. This can be done on stage. (Laura/IT)
• December 14: Push out Socorro 1.7.5.4 which adds support for UTF-8 encoded e-mails (Laura/Socorro/IT)
• December 15: Send manual e-mail to users experiencing the Russian crash and use this as a basis for establishing additional requirements for Phase 2 in Q1 2011. (Michael/David/Austin/Christian)
• December 17: Follow up with Daniel to determine hit rate (David)
• December 17: Send out e-mail to the remaining Russian users
• December 23: Finalize PRD to include requirements for Phase 2 in Q1 2011 (David)
• Ongoing - Next Firefox 3.5.x/3.6.x release: Complie a list of actual crash bugs fixed (Christian)
• Ongoing - Next Firefox 3.5.x/3.6.x release + 7 days: Send out first "crash fixed" e-mails (Michael/David/Christian)

Future requirements

• Ability to define a campaign with multiple crash signatures, each with its own set of filtering restrictions (product, versions, etc)
• Ability to wildcard Firefox versions (e.g. "3.5.*,3.6.*")
• Ability to specify whether a campaign is for a fixed or ongoing crash issue (which determines whether this is the final e-mail campaign for those users, or if we are allowed to send additional information in the future)
• Asynchronous sending of the e-mail -- currently you have to select a really small date range because otherwise the system times out. If this could be changed to e.g. a backend query/job, sending e-mail in a large campaign would be much more straightforward.

Questions/Discussion

• Some crashes have more than one associated bug. Should we email the user each time a bug is fixed or when all the bugs are fixed (or some other model)?
  • the period of when bugs get fixed to when the crash happens might be very long. bugs being fixed on the trunk now might not be available in a final release until end of q3 or q4. It doesn't make much sense to e-mail users now about fixes that won't appear in a final firefox 4.0 until then, or e-mail them at the end of the year about that that crash they had 6 months ago. A single signature with many bugs makes it nearly impossible to correlate if any one of the bugs might have fixed the specific bug that the user just crashed on. we would need to start storing the full stack trace(s) of the bugs that we have fixed and map them to the full stack trace that the user experienced. -chofmann

• What volume of users supply emails? (Check: For each of the topcrashers, how many associated user emails do we have?) The architecture of the solution for sending email depends on the volume.
  • Also depend on what events we send e-mails for. (see below: expect to send about 12k e-mails per day for just incoming reports where we are auto responding when a user sends us an e-mail address, and around 5k per day if we e-mail when there is a bug on file associated with the signature) -chofmann
  • Do repeated crashes mean repeated e-mails to the users with the same message, or do we track so that we don't keep telling users there is no fix for their crash yet? if we don't track it the e-mails start to look like spam. --chofmann

• we could be over engineering things here. the system that we had with talkback served a good purpose. when we found a specific crash that we had something specific to tell users about, we constructed the message, then we queued up the system to watch for that signature and send the message. This was for a very small number of the total overall crashes, but had useful information as opposed to just auto-responding. e.g. "from your recent crash we detected that you need to upgrade to Skype X.X to fix the crash."
  • For the record, this is pretty much exactly what this project is about: reaching out to users when we actually have something meaningful to say (like "your crash has been fixed").

At any particular point in time we might have something this specific to say in an extremely low pct. of cases. Numbers below indicate this might be lower than 1% of the time a user reports a crash to us.

• sending e-mails with technical instructions on how to fix or mitigate crashes, and espcially instructions that involve downloading and installing software is open to impersonation, and phishing. we need build in ways for users to authenticate that the message came from mozilla and that the same instructions are available on an "offical" mozilla hosted website.

• <bsmedberg> We need to be very careful with the about:crashes changes in order to preserve user privacy. The webservice calls have the ability to correlate a user with multiple crash reports: we probably should not perform them automatically, and should probably avoid sending any cookies over the wire when we do make the requests.

some numbers

 
 14k   -count of reports that have e-mail address, 
378k   -total crashes per day, 
  3.6% -ratio of e-mails to total crashes
  5k   -signature has e-mail and an assigned bug 
  1%   -ratio of signatures with emails and assigned bugs to total crashes
 35%   -ratio of reports with e-mails to reports e-mails with bugs.

                                             email& email&
date           email=yes #crash email/crash   bugs  bugs/tl   email/email&bug
20100530-crashdata.csv 12596 341229 0.03691   4319  0.0126572   0.342887
20100531-crashdata.csv 13567 372037 0.03646   4784  0.0128589   0.35262
20100601-crashdata.csv 13941 378282 0.03685   5053  0.0133578   0.362456
20100602-crashdata.csv 13806 379287 0.03639   4793  0.0126369   0.347168
20100603-crashdata.csv 12353 332924 0.03710   3855  0.0115792   0.31207

-chofmann

• The 3.6% might be heavily affected by the fact that the checkbox is unchecked by default. I.e. an insensible default. Can we change that?