User:Standard8/Thunderbird Automated Tests Planning

From MozillaWiki
Jump to: navigation, search

Where we are

We have two sets of automated tests:

  • xpcshell-tests
  • Leak and Bloat tests

More info MailNews:Home_Page#Automatic_Testing.

We are also starting to write MozMill tests. At the moment we have a few tests checked into comm-central plus a runtest.py that will run those tests within Thunderbird if you have built Thunderbird in a dev environment.

The Plans

Performance

xpcshell-tests

xpcshell-tests are now moving along nicely. As we have leak stats as well, there isn't much more to do, except for fixing the remaining reported leaks and writing new tests as we fix bugs.

I think expanding facilities such as fake server will most likely happen as there is demand for it.


MozMill and Leak & bloat tests

We have a great opportunity here to start automated UI testing, something which we haven't been able to do up until now.

The rough plan is this:

  1. bug 458352 support for different profiles when running tests
  2. bug 500142 Integrate mozmill into the build system - tests can be run with "make mozmill" and "make mozmill-one"
  3. (in progress) bug 504304 Test VM/buildbot slave for initial results and automation. Aim is to a) check the stability, b) use as a platform for implementing and testing the necessary mozmill test support infrastructure, c) get initial results alongside builds on day-to-day basis. ThunderbirdTest tinderbox
  4. Integrate httpd server into mozmill run scripts - mozmill actually already has a httpd server integrated. See example on bug 492279
  5. bug 500753 Integrate fake servers into mozmill scripts.
  6. Bug TBD. Once support infrastructure is stable, expand mozmill tests across platforms.

Future:

  • Possible check-interactive target
    • Set up profile, but don't actually run via mozrunner, just run up TB with the profile.
  • bug 506202 Hooking Mozmill up to make package-tests.
    • This would allow non-devs to download a test package and run alongside builds. Could also move mozmill tests to a separate slave if required.
  • bug 458352 Hooking Mozmill up to Leak and Bloat tests.
    • This step currently blocked by bug 500201 - MozMill is leaking lots of items. Hence moved to end.
    • Once we've got that support, I'd like to replace the existing Leak and Bloat tests (which currently use js files installed in chrome) with a mozmill version. This will help with the extensibility of the tests and give us a better base for which we can start extend those tests bug 458351.

Suggestions:

  • A pass count of zero should return failure.
  • It would be neat if we could return the test results via a JSON blob somewhere rather than the current tradition of text-mangling. I am mainly thinking in terms of having a web UI that is better able to tell us where things went wrong. But that's just pipe-dreams and tinderpushlog with its magic already gets us more than 80% of the way towards usability...

Intermittent MozMill Failures

Note: asuth killed most of these failures. We've only one or two left now, I'll update this next time we see them.

Retrieved from "https://wiki.mozilla.org/index.php?title=User:Standard8/Thunderbird_Automated_Tests_Planning&oldid=162754"