Overview

• Help Wanted

Dave Liebreich is interested in creating running code to test the mozilla codebase. Not just end-to-end automated tests, but unit tests, module tests, test harnesses, test reporting infrastructure, and anything else that lowers the barrier to creating new code that gives someone feedback.

Dave needs help. The codebase is huge. The community is huge. The task is huge.

This page is the first cut at collecting Dave's plans and investigations.

Guiding principles

• Testing is good
• Testing provides technical information about the product/system under test
• There are lots of different types of testing, each providing a different type of information and involving different trade-offs
• replacing a test that is useful to person A with a test that is useful to person B but not useful to person A is bad - just add the new test
• automated developer testing works only if the developers are writing the tests.
  • if developer finds something that is difficult to test, then developer can make code more testable
  • if code is more testable, then tests can be more powerful
• feedback from user community is unstructured and somewhat unpredictable, but incredibly useful. We should not extinguish this valuable channel of information.
• developer unit tests are typically not-very-powerful tests. The value comes from
  • immediate and precise feedback - written close to the code
  • there are typically lots of them
  • they function as executable specifications for the code, and can replace some documentation and usage examples
• we need involvement from toolsmiths and developers and testers
  • toolsmiths maintain the infrastructure, and find/adapt/use tools to solve problems
  • developers write developer tests
  • testers write end-user tests
  • each group shares domain knowledge with the others

Projects

• Catalog all existing test suites and/or test cases
  • special page for security tests
• investigate existing testing tools
  • Watir (possibly using JSSh)
  • xulUnit, jsUnit, jsUnit (yes, there are 2), xUnit, FIT
• create test infrastructure architecture
  • common invocation and reporting formats for all tests
    • or choose 1 of no more than 3, if 1-size-does-not-fit-all
  • put docs on wiki, code in cvs
  • follow list of principles for automated testing
    • data-driven (data separate from test code)
    • Keyword driven
      • action keywords
      • fixtures to implement action keywords
    • focus on oracle (how do we know the test passed or failed)
    • do not rule out computer-assisted manual testing (no automated oracle)
    • use model-based test scripting when possible
      • allows for high volume automated testing
      • can walk the state transitions and detect failures other than asserts or crashes
• Community outreach
  • identify people in the mozilla community who are already test-infected
  • identify test-infected people outside the mozilla community and ask them to help
  • hit up grad students who can make this work part of their academic projects

repository structure

• mozilla/testing
  • top level for common testing tools, harnesses, etc
• mozilla/testing/harnesses
  • moved from mozilla/tools/test-harnesses
  • code we develop to run tests
• mozilla/testing/thirdparty
  • code from other projects that we use, and that we want to snapshot
• mozilla/testing/suites
  • test suites (collections of tests and/or test suites)
• mozilla/testing/tests
  • tests that do not fit anywhere else (yet)
• _some-code-dir_/tests
  • tests for the code/function/feature

how to add test cases

• basic types of tests
  • "make check" targets
  • jsunit tests
  • layout comparison tests
  • jssh driver tests
• typical types of tests
  • make check
    • does not need entire browser instance
    • stand-alone executables
      • unit tests
      • xpcom consumers
      • xpcshell-based test harness tests
        • interpret tests written in js
        • helper functions exist to implement event loop
    • if exit is non-zero OR test prints FAIL then test fails
  • jsunit
    • runs in content in browser
    • supports asynchronous tests via semaphore variables (run test when test_done -> true)
    • good for DOM tests
    • added code to print results to stdout
  • layout comparison
    • compare layout/rendering of two different pages using canvas
    • dynamic generation of reference image
    • good for layout/css tests
      • <span style="color: rgb(0,255,0);">This text is green</span>
      • something much more complex that should paint green text
    • not sure of pass/fail indicator yet
  • jssh driver
    • uses jssh as "remote control" of browser, with chrome privileges
    • good for jsunit-style tests that need to muck about with preferences
    • good for tests that should quit or crash the browser
    • not sure of pass/fail indicator yet
• weaknesses
  • make check
    • getting the linking right for standalone executables is hard
    • need to explicitely instantiate all services/components needed
  • jsunit
    • can't run chrome stuff
    • can't deal with modal windows
  • layout comparison
    • can be a lot of work to generate test cases
  • jssh driver
    • only works with modal windows on trunk (1.9)
    • slow