Mobile/Testing/10 14 15

From MozillaWiki
Jump to: navigation, search

Previous Action Items

Status reports

Dev team

Rel Eng

  • Running chunked Android debug (jsreftest/crashtest) tests on Try to generate data for SETA bug 1201236. Will enable reftests later to generate data as well while keeping pending counts low.

A Team

  • android-2-3-armv7-api9 Total failure rate: [0.86%]
    • last week 3.11%
    • total failure excluding retries 0.81%
  • android-4-3-armv7-api11 Total failure rate: [2.02%]
    • last week 1.68%
    • total failure excluding retries 1.45%

Data fetched for: 2015-10-06 09:54 — 2015-10-12 22:18

  • [gbrown] Working on:
    • bug 1087791 Add |mach reftest| and |mach crashtest| for mobile/android
    • bug 1197716 Package fonts separately for reftests

Autophone

  • Phonedash vs. Perfherder

Perfherder will only support Autophone Talos. We will revisit improving Phonedash for S1S2/Webapp performance tests. Since any noise in the device's behavior or the framework tend to increase the measurements, will investigate using the "Greatest Lower Bound" of a measurement set as an indicator of the "Best Measurement" possible which will hopefully be a better indicator of regressions than the mean.

  • Regressions
    • Bug 1213175 - Autophone - Throbber start regression Android 2.3 Remote Tests 2015-09-28 on mozilla-inbound
    • Bug 1209653 reversed a framework caused improvement which occurred during the 2015-08-21 deployment of Bug 1156812 - [mozdevice] Support specifying alternate log buffers when retrieving or clearing logcat
    • Bug 1212995 - Autophone - Throbber stop regression 2015-10-07 on fx-team (Fixed by backout)
    • Will investigate trend of death by a 1000 paper cuts
  • Deployments
    • Bug 1209796 - Autophone - 2015-10-04 deployment
      • Bug 1208782 - Autophone - do not attempt to get response json during Treeherder submission error if response is None
      • Bug 1209647 - Autophone - eliminate startup check for network connectivity
      • Bug 1209651 - Autophone - do not allow logcat device error to prevent setup_job initialization
      • Bug 1209653 - Autophone - after clearing logcat, specifying -b main can hang
      • Bug 1209675 - Autophone - Logcat should use PhoneTest loggerdeco
      • Bug 1209691 - Autophone - handle incorrect logcat dates emitted by devices.
      • Bug 1211227 - Autophone - work around webapp Crash Reporter moving dmp files
      • Bug 1211228 - Autophone - prefer "Start proc" over Gecko related messages when determining start time
      • Bug 1211229 - Autophone - reduce stderrp_accept to improve reproducibility
    • 2015-10-05 Deployment
      • Bug 1211655 - Autophone - Ensure adb.py methods propagate root to other called methods
    • Bug 1212757 - Autophone - 2015-10-09 deployment
      • Bug 1212758 - Autophone - ADBDevice.mkdir - reproduce behavior of mkdir -p
      • Bug 1212766 - Autophone - make webapp profiles.ini readable and handle failure to obtain the remote profile path
    • Bug 1214110 - Autophone - 2015-10-13 deployment
      • Bug 1211252 - Autophone - Unit Test not (un)installing robocop.apk
      • Bug 1213866 - Autophone - adb.py - add a logcat cleared message to the logcat whenever it is cleared so it is never completely empty
      • Bug 1213868 - Autophone - catch device logcat exceptions in PhoneTest.setup_job
      • Bug 1213879 - Autophone - work harder to create clean profile directory during PhoneTest install_profile
      • Bug 1213922 - Autophone - call power_on after reboots and before s1s2 iterations
    • 2015-10-13 Deployment
      • Bug 1214273 - autophone - 2015-10-13: deploy nexus-7 for try only and tp4m/tsvgx-remote on mozilla-inbound
      • Bug 1192069 - Autophone should submit data to perfherder
    • 2015-10-14 Deployments/Issues

We had issues with the Talos tests due to some problems with pushing the test files to the device. While attempting to work around the issue by upgrading adb to a more recent version, other issues appeared. Android Platform Tools 23.0.1's adb (still at 1.0.32 version) failed with syntax errors on repeated wait-for-device in command lines and was subject to multiple, frequent disconnections. Android Platform Tools 23.1 rc1's adb (still at 1.0.32 version!) also suffered multiple disconnections resulting in very slow job throughput. We are in the process of recovering. Hopefully Linux will solve this, but we'll know more later.

Perfherder

Round Table

  • tcheck on talos is very noisy, trying to figure out what to do with it in bug 1213032
    • in addition it is the test with the highest chance of failure on autophone- when we switch devices it is 50/50 if it will work or not

Action Items

  • snorp - investigate when to disable the webappstartup test. (bc will send email)
Retrieved from "https://wiki.mozilla.org/index.php?title=Mobile/Testing/10_14_15&oldid=1100621"