Proposed Mobile Testing Support Planning and Timeline

The following information was gathered from various meetings and email exchanges that took place between the Firefox on Android product team, IT, and ateam. This document should eventually live with the product road maps for our various mobile offerings, but since those road maps are currently being redone, it is documented here as a placeholder.

Android 2.x Support Assumptions and Goals

Current User Populations

• [Kevin] In March Google estimated the market share at 7.6% of the devices that connected to the Play Store during a 14 day period. Google recently updated their data and 14 days ending May 1 Android 2.2 was 3.4% of devices that connect to Play.
• [Karen] 2.2 represents 4% of our current Firefox user population. 2.3 represents 23%. We recently expanded ARMv6 support and have some programs on the books to raise awareness of Firefox for these devices. The suspicion is that many will be on 2.2 and 2.3 versions thereby providing growth on those two OS versions.

Current Support

• [Amy] We have about 320 tegras running Android 2.2 that perform tests for 2.2.
• [Brad] Mozilla deems that Android 2.2 tests provide some acceptable test coverage for 2.3 (even though we are not running 2.3 on the hardware).
• [Brad] We perform some manual QA for 2.3.
• [Brad] The nature of the bugs we see that are on 2.2 but not 2.3 are that we're using APIs not present on 2.2. These bugs cause crashes that we need to catch in CI.
• [Brad] Bugs we see in 2.3 and not 2.2 tend to be minor UI styling differences, which CI doesn't tend to catch anyway. So standing up 2.3 on hardware while doing 2.2 manual testing is not a good option.

Proposed Future Support

• [Brad] Getting 2.3 automated testing is a requirement for decommissioning 2.2.
• [Clint] We would like to use the pandas to stand up 2.3, but the work to accomplish that won't (as currently scheduled) start till Q4 2013 and will probably take 7-9 months.
• [Clint] Linaro has officially EOLed their 2.3 support, but they pointed us to their last known good build of Android 2.3.7 for pandas. If we find OS stability issues, we'll explore contracting Linaro to fix their builds.
• [Karen] There will never be a request to test any version of Android on ARMv6 hardware from Fx for Android because it is cost/time inefficient.

Android 4.0 Support Assumptions and Goals

Current Support

• [Amy] We currently have pandas running Android 4.0 and performing tests for that OS.
• [Joel] Now that the issues with the Android 4.0 tests have been fixed (much recent work done), pandas are showing on par success/stability rates with the tegras. These range of these results change regularly and vary anywhere from 14% to 4% failure rate. They are reported each week at the Android mobile meeting.

Proposed Future Support

• [Amy] We intend to continue to provide Android 4.0 support on pandas. Relops is working on a higher density design that will double our capacity footprint for the boards.
• [Kim] Releng is working on migrating Android 4.0 pandas to use mozpool.
• [Amy] Various vendors are also starting to enter the ARM market and plan to start offering solutions in the next year or so which we will want to investigate as replacements for our in-house built gear. It's not known yet whether these offerings will be ARMv7, ARMv8, etc.

Android x86 Support Assumptions and Goals

Current Support

• There is no support currently

Proposed Future Support

• [Clint] Ateam is working on standing up Android x86 support on emulators because there is no hardware boards at this time for Intel's android chips that are released
• [Clint] Eventually there might be boards/hardware that we want to stand up, but that's far in the future and a large unknown.
• [Amy] Since it's x86, I think we have some hope of buying an off the shelf solution for that instead of having to build our own.

Firefox OS Support Assumptions and Goals

Current Support

• [Amy] We perform automated testing on emulators in AWS as well as on some Fedora Mac minis (the latter will be going away in a year when we close SLC1 or sooner).
• [Amy] We have panda boards designated for Firefox OS use, but they are not being used/watched.
• [Clint] Developers do not have adequate access to pandas to debug/write tests and we can not hook the Firefox OS pandas up to Try because it's hard coded to use hg whereas the Firefox OS source code is stored in git.
• [Clint] We have a small number of phones for automated testing (but not hooked up to and possibly closing the tree).

Proposed Future Support

• [Clint] At some point (sometime in 2014?) we hope to resolve the issues with Firefox OS on the pandas and start using them for hardware tests.
• [Clint] In order for pandas running Firefox OS to become a tier 1 platform, our platform team will have to support the Firefox OS panda kernel. Currently this support has been de-prioritized in favor of stabilizing and supporting real phone hardware. Ateam will help the core platform team with this in future quarters, but will never have the scale to completely own the kernel. That has to come from development.
• [Clint] We therefore have requirements to grow the number of automated testing on physical phones (probably 100 or so at a time) over the next year. Solutions for this may include building our own hardware management solution, looking at third party offerings (phones in the "cloud," outsourcing to humans or robots, etc). As of now, this is an unsolved problem.
• [Clint] The restrictions around obtaining the development phones, combined with the high rate of change on the devices/build systems and the legal restrictions on the proprietary codecs that are required to actually build a functioning phone would likely make outsourcing horribly inefficient.
• [Amy] We most likely don't have the necessary expertise/man power to build a operationally maintainable physical phone solution in-house, so if we go that route, we should investigate outsourcing the construction of such a system to a third party who is versed in the necessary technologies.
• [Amy] The HD panda chassis can be used for Firefox OS in the future if more capacity is needed. We will also be looking at those third party offerings to replace our custom-made rackmount chassis.

Proposed Schedule

In each of these phases relops/ateam will be looking for off the shelf third party solutions (whether in house hardware or services in the cloud) for both Android and Firefox OS testing support. Many vendors are entering the ARM server space, but it remains to be seen if any of those offerings will be applicable to mobile testing (so far, we have seen none with GPUs).

By the end of 2014 it's hoped that we will be consolidated onto one datacenter hardware platform (pandas) with some virtualization in AWS for tree-driven testing of all mobile platforms. The caveat is if we find an off the shelf x86 or ARM solution that works better/is easier to support than the pandas. We also expect to have some number of physical phones in a non-datacenter environment.

2013 Q2 & Q3

• MILESTONE: Ateam focuses on getting x86 emulation working.
• Ateam investigates solutions for physical phone testing for Firefox OS.
• MILESTONE: Relops focuses on prototyping and testing the HD panda solution.

2013 Q4

• Ateam begins the discussion with engineering about supporting the panda 2.3 kernel.
• Ateam and relops focus on getting Android 2.3 working on pandas, possibly contracting with Linaro to request modifications to their OS implementation.
• MILESTONE: Ateam selects (and contracts to implement?) a solution for physical phone testing of Firefox OS.
• MILESTONE: IT focuses on moving tegras out of MTV1 (how we do this is TBD and currently under discussion).

2014 Q1

• Ateam and relops continue working on Android 2.3.

2014 Q2

• Ateam and relops continue working on Android 2.3, possibly switching focus to Firefox OS on Pandas if we complete Android 2.3 (with the possible assistance of releng and maybe more of IT for help with hg vs git?).
• MILESTONE: Relops decommissions SCL1 and all Fedora mac minis running Firefox OS emulation tests.
• Product starts communicating the EOL of Android 2.2 support.

2014 Q3

• MILESTONE: Android 2.3 tests run on pandas.
• Ateam and relops/releng/IT continue work on Firefox OS on pandas.
• Product continues communication about the EOL of Android 2.2 support.

2014 Q4

• MILESTONE: Firefox OS hardware tests run on pandas.
• MILESTONE: Android 2.2 hardware (tegras) are decommissioned and OS support is EOL.