Mobile/Testing/01 08 14
From MozillaWiki
Contents
Previous Action Items
Status reports
Dev team
Rel Eng
- Moved tegra staging monday -- runs look good
- Tegras down for 48 hours this weekend (starts with TCW at 9am PT on sat)
- Hope is for them to be up as of Taiwan Monday Morning, and hopefully without any issues.
- TCW includes move of the bm-remote servers, used for talos on Android, including for pandas.
IT
- Staging tegras have been moved to SCL3
- Production Tegras are moving to SCL3 this weekend (Jan 11,12)
A Team
- bug 929447 Enable panda reftests, ride trains
- patch reviewed and ready to go
- snatched failure from the jaws of success (new oranges on Cedar this morning)
2.3 emulator automation
- bug 910092
- GLES2 problems with emulator running Android 2.3 as seen at https://bugzilla.mozilla.org/show_bug.cgi?id=910092#c174 - #c178
- graydon has emailed the Google engineer who did the GLES2 backport in 2011...no reply yet
- graydon is trying other build configurations
x86 automation
- All tests running hidden on Cedar, with some failures
- S4 failure rate is now < 4%: https://tbpl.mozilla.org/?tree=Cedar&showall=1&rev=36c8eeaf1589
- Biggest remaining issue is intermittent loss of connectivity: bug 927602
Autophone
- Nexus Mochitest-Canvas testing
- Nexus 4 and Nexus 7 2012 running and reporting to brasstacks: http://brasstacks.mozilla.com/autolog/?tree=mozilla-central&source=autolog
- Ok to order Nexus 5 and Nexus 7 2013 devices?
- Comparing autophone s1s2 to eideticker, leads me to believe that autophone s1s2 adds value and should be kept but it should be improved to better meet our needs.
- We should evaluate the test files in s1s2 and to see if they are appropriate.
- Are the throbberstart, throbberstop messages meaningful to measure? Should we be measuring something else?
- Experience dealing with older regressions leads me to believe that large regressions are easier to spot even when grouped together in merge changesets but that multiple small regressions are much harder to spot and isolate especially when they occur in the same merge. These small regressions accumulate over time. See http://gbrownmozilla.wordpress.com/2014/01/07/firefox-for-android-performance-measures-2013-in-review/ and especially bug 916099.
- bug 904088 Autophone - s1s2 - Regression in "time to throbber stop" on Aug 9, 2013
- bug 909380 Autophone - s1s2 - Regression in "time to throbber stop" on Aug 24 2013
- bug 916099 Autophone - s1s2 - Regression in "time to throbber stop" on Aug 28 2013
- If we agree that Autophone s1s2 should be kept, then the next steps should be:
- bug 957312 - Autophone - add repository to db schema.
- bug 957360 - Autophone - change phoneid to have hardware as most significant.
- Should run tests on fx-team, mozilla-central, mozilla-inbound.
- This will help identify the original regressing changesets on the original repo without having to wait for the merge and without having to deal with a huge merge changeset.
- Developers should track autophone for their builds on the appropriate repo for regressions.
- We may have to reduce the number of iterations from 16 and possibly partition repos across devices.
- Improve bisection processes for isolating regressing changesets. http://inbound-archive.pub.build.mozilla.org/ will be helpful for triaging regressions after May 2013 for mozilla-inbound, Aug 2013 for mozilla-central and Sept 2013 for fx-team.
- Being able to hide revisions which have been backed out may be helpful in reducing the visual noise in the graphs.
- We should use tracking bug 953342 so we can easily find issues which have already been reported.
Eideticker
- Not much new to report other than the usual maintenance work
- "Should run startup / load tests against Chrome weekly on Galaxy Nexus" (bug 943500) still planned, just haven't had time yet.
Round Table
- trobopan has so much [noise], should we continue to run it?
- Callek mentions that trobopan has a lot of failures as well: bug 956205, bug 950031, bug 935911, bug 934968, bug 931321, bug 930954
Action Items
- blassey to report back on his meeting this Friday, re: future of Pandas, etc.
- bc ran out of time for autophone...he'll finish up next week