B2G/QA/Meetings/Automation/2014-09-04

Overview

This is a weekly meetup to discuss all things regarding B2G automation and Quality.

This week we had big issues with the builds after the merge on 2 Sep. There are still a few outstanding issues: (only affects trunk - 2.2)
- Bug 1062192 - Random tests are failing with 'JavascriptException: TypeError: this.dataStore is null' error
- Bug 1062255 - Crash prevents any further tests from running
- Bug 1062902 - Unable to start Marionette session on freshly flashed device
Started to run automation and report on v2.1 as master is now v2.2
2.0 CS - sept 15th? - we can stop smoketest automation after this date

[Tony] bug 1055593 had the wrong regression range, and backout didnt fix the problem in bug 1062192. how could we investigate this much better?
- due to intermittent issues, its hard to find the regression range. investigative work showed the browser migration bug was the last commit in the range. We dont know what is causing the problem.
- trying to run this locally was still intermittent, needs 15-20 testruns before hitting it
- trying to fix this bug would have hit the other crash bug 1062255. required restarting the whole testrun
- option 1)do what we did above, it was the best steps to identify the cause. asked for a backout, and retest. unfortunately, it wasnt the culprit
- option 2)try the backout patch locally, and see if it worked. but given its intermittent, it could still not solve the problem. TODO: have a backup person that can do the backout/regression testing
  - Cons: this is very time consuming. use a local build, and investigative measures
- option 3) turn off the intermittent test suite, and have people investigate the issues.
Should we have a non-smoke tinderbox job so that we can offer sooner a regression window for the corresponding bugs
Bugs that are fixed after the testrun (jenkins one) and we still report them
Where can we find documentation or specifications on how apps should work and what is the expected behaviour email qa-b2g-internal@mozilla.com and ask the Functional QA owners or the UX, spec, user stories[Done]

[jsmith] Send out broad email to b2g-internal about automation blockers & implications (Done)
[Tony] follow up with bug 1061723 on why james is recommending full flash
- Also see https://bugzilla.mozilla.org/show_bug.cgi?id=1051070 (perf issues/fixes only exposed on full flash)
[Bebe] rerun the trunk automatoin with rebasing v123 on Flame and shallow flashing tip of gaia/gecko. and see if the bugs above still reproduce

(Done)