Mobile/Testing/06 19 13
From MozillaWiki
Contents
Previous Action Items
-
(jmaher) File bug to get access to foopy and some pandas or tegras to further investigate bug 877779 -
(dminor) File bug to run Panda reftests on Cedar -
(kmoir) Document process for getting rogue pandas and tegras removed by builddutyIt's easier just to ask who's on buildduty to open a bug but here's a link to our bugzilla machine management component where you can just ask for a specific device to be disabled.
Status reports
Dev team
- Working on
- bug 883539 Frequent Android 4.0 rc2 retries: "Remote Device Error: unable to connect to %s after %s attempts" % (self.host, retryLimit)
- bug 882932 Remote mochitests should invoke the httpd.js from hostutils, not from the changeset
Rel Eng
- kmoir will land final changes for non-talos android mozharness/mozpool tests early next week when she returns from PTO. Tests are green in staging and patches have been reviewed. bug 829211
IT
- Still working on a higher density chassis. Waiting for DCOps to rack in Data Center. - https://www.dropbox.com/sc/j5fpbwwd7ki2knb/fPQplsqiya
- Tegra move - mrz is working on a proposal to present to the team for review
- Draft https://mana.mozilla.org/wiki/display/~mzeier@mozilla.com/Tegra+-+Migration+out+of+650+Castro
- There has been email back and forth on this, nothing is finalized yet.
A Team
tegra failure rate [8.68%]
- tp, r3, j1, rc2, m3 <- all >15% failure rate
panda failure rate [19.62%]
- rpr, ts, tpn, rc1, rc2, m7, m2 <- all >15% failure rate
top 5 orange factor android bugs:
- bug 807230 - Intermittent DMError: Automation Error: Timeout in command {ls,ps,isdir,mkdr}
- dminor working on this, most common on pandas, but across all mochitest/reftests. NOTE: not seen on talos!!!
- bug 711725 - Tegras and Pandas disconnect with "remoteFailed: [Failure instance: Traceback (failure with no frames) ...
- callek attempted fix, no luck, large majority on tp4m talos test, rarely on other suites
- bug 874972 - Intermittent Android retry "Remote Device Error: updateApp() call failed - exiting"
- a newer issue, tegra only! Only occurs on a handful of naughty tegras.
- bug 817024 - Tegras/Pandas disconnect with "remoteFailed: [Failure instance: Traceback (failure with no frames): ...
- similar to other one, but specific to inside of test case, mostly on tp4m
- bug 845162 - Intermittent Android timeout in test_replay_metadata.html,test_playback_rate.html...
- m2 on pandas, possibly a root cause in a test case early on causing OOM or other state changes?
Panda Reftests (dminor)
- Got agreement on syntax - will retest patch today and put up for another review once it looks good locally
x86 automation
Autophone
- New production site: http://phonedash.mozilla.org
- Locally increasing devicemanagerSUT.reboot_settling_time to work around slow sdcard performance on nexus ones.
- No progress on getting reftest unittests running.
Eideticker
- LG G2X seems to have been dying on the "imgur" test. Worried it might be memory exhaustion or something. Filed bug 884907 to investigate.
Round Table
- (gbrown) What can we do about bug 883539? Can we update the Panda image?
- (jmaher) we could update the image (possibly with a new watcher?), is there other hacks we could make as well?
- (jmaher)watcher update?
Action Items
- Joduinn going to follow up with bmoss about his ideas on the tegra move decision
- Jmaher and team going to keep digging into the failure rate spike, potentially rolling back the recent sutagent change
- Bc will have an update on autophone reftest testing next week
- callek has on his plate this week to write up a concise plan to update the watcher across the board.
- gbrown - will start on making a new image for the pandas to address the retry bug, and then we can use mozpool to distribute it.