FirefoxOS/TestingInfrastructureImprovements

Meeting notes Aug 17

Marionette JS vs. Python

Original MarionetteJS duplicated a lot of the python infrastructure. Including parts that we've worked a long time on to stabilize and get running smoothly. Since then work has been done to make MarionetteJS reuse more of the existing python-based infrastructure.

However there's more to be done. MarionetteJS tests can almost run on-device, but only python tests are able to run in the twice-daily QA automation. We also can't extract crash dump from MarionetteJS test failures, nor gecko logs or screenshots.

Action Items

• Stand up MarionetteJS to run in the existing twice-daily automation.
• Enable turning on MarionetteJS tests only on specific platforms (on-device vs. emulator vs. Mulet)
• Try running a few MarionetteJS tests in the twice-daily automation and weed out stability issues.

Mach integration

It is currently very hard for non-FirefoxOS mozilla developers to locally get all dependencies and run our test suites. This is needed in order to help non-FirefoxOS mozilla developers to fix any test failures and regressions that they cause in FirefoxOS.

For Firefox desktop and Fennec we use mach to solve some of these problems. We should potentially look into integrating with that. However we didn't end up having time to get into enough details here, and it seemed lower priority for now.

No action items for now.

Meeting notes July 28

Intermittent failures

We still have problems with intermittent failures. In particular, we still have a few infrastructure issues which is causing intermittent failures in *all* Gij tests.

Some of these infrastructure issues are due to known problems that are being worked on, Aus is on top of that. But the big one is that we still have a problem with the socket connection randomly disconnecting. This is a really tough problem and people have worked on it for years with no progress.

These intermittent issues are somewhat less of a problem because we now run tests three times and only if all three fail will we consider it a failure. Unfortunately this means that bugs in our product which only happen intermittently are no longer caught.

We also have some intermittent issues in individual tests, but unclear how much. One proposed solution was to write helper functions which help with writing stable tests. The "Page object pattern" used by calendar tests were pointed out as a good example.

Action items

• Fix the known causes of infrastructure issues (Aus is already working on this).
• Catch the socket disconnect issue in rr.

Get more info when a test fails

When a test fails, we would like to collect

Gecko log (bug 1188648) Error stack Screen shot crash dump in case of crashes

Action items

• Add the above to Gij infrastructure

Which platforms do we want to maintain

We currently have too many development platforms: mulet/gaia-in-nightly/b2g-desktop/simulator/emulator-ICS/emulator-L/B2GDroid/on-device.

We should narrow it down to: Mulet, Emulator-L, B2GDroid (not yet ready), and on-device.

B2G-desktop, gaia-in-nightly and simulator should be removed from the tree and from test automation. But first we need to make sure that all tests that are currently running on B2G-desktop are also running in Mulet.

We also want to kill |DEBUG=1 make|. But first we need to improve the "shift-f5 reload" development flow. Possibly by hooking it in to a super-fast rebuild step.

Action items

• Ensure that all tests that are run in B2G-desktop is also run in Mulet (Lissy is working on it)
• Kill B2G-desktop, gaia-in-nightly and simulator
• Stand up Emulator-L (Hsin-Yi is working on it)
• Figure out how to make "shift-f5 reload" work (Tim is thinking about this, no one is working on this yet)

What tests do we need to run on-device/in emulator

Running on emulator is really slow. In part because of arm. In part because we restart emulator. We would like to run most tests in emulator (except unit tests). However we need Emulator-x86 for that to be viable.

We have tests for Bluetooth/RIL/Wifi/NFC which run on emulator, but they are not reporting to treeherder because we only have emulator-ics on treeherder and it doesn't support those features.

We are running perf tests on-device, but no one knew if they are reporting to raptor.

Action items

• Stand up Emulator-x86 and Emulator-L on treeherder.
• Enable Bluetooth/RIL/Wifi/NFC tests on treeherder.
• Make sure perf tests that are running on hardware report to raptor.