User:Rcampbell/Toronto Summit Notes

From MozillaWiki
Jump to: navigation, search

These are robcee's bulletized notes from the Toronto QA Development Summit on January 25th-26th, 2007. They are rough and there are large pieces missing. Feel free to fill in the gaps.

automated testing

Buildbot functions

  • pull nightly
  • initiate test runs
  • build given patch
    • Ben Hearsum doing work with Build team on this
  • select platforms
  • reporting handled by performance testing framework
  • litmus reporting
  • clean up build directory/create cleaner mirror
    • get rid of kruft
  • drive performance testing

Reporting Tools

Survey of reporting tools

  • litmus
  • buildbot
  • tinderbox

Report aggregation

  • RSS
  • Atom
  • Microformats
  • Security
    • passworded version
    • email
    • irc
  • report drill-down
    • history
    • test details
    • logs
    • store links in database for detailed analysis

Rollups

  • graphserver
  • buildbot?

History

  • store some set period
  • retain important historical milestones / benchmarks

JS Tests

  • specialized builds
  • buildbot vs tinderbox canned builds
    • requires js shell, xpcshell, can't use 'off-the-shelf' versions.

Metrics

  • bug density
    • per component
    • map bugzilla modules
    • checkins churns
  • lines of code
  • performance thresholds
    • links on wiki
  • top-crashers