QA/Test Automation/2010-11-17

From MozillaWiki
Jump to: navigation, search

previous meeting | Meetings | next meeting »

Dial in

 # 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL)
 # 1-800-707-2533 (pin 369) Conf# 315 (US)
 # irc.mozilla.org #mozmill for backchannel

Attendees

  • Cameron, Geo, Al, Matt, Anthony, Aaron, Adrian, Henrik, Dave

Last weeks action Items

Goals Overview

Risky Goals

Project Status

Tests / Shared Modules

General (Henrik)
  • Waiting for Clint's proposal how to do Drag&Drop - needed for Panorama tests
Broken Tests
  • Firefox 4.0 (Geo)
  • Firefox 3.5/3.6 (Anthony)
    • Landed waitForTabPanel() to make notification bar handling more reliable (5 of top-10 fails resolved)
DOM Walker + L10n API (Adrian)
  • final review stages
Redesign Shared Modules (Geo)
Test Refactoring (Anthony)
  • Draft styleguide being composed (targetting EOW)
  • Begin refactoring once styleguide approved
Local Test Data (Anthony)
  • No progress yet - considering community outreach to create testpages
    • 35 tests are already local
    • 4 tests remain which can be made local
    • remainder are not doable in Mozmill
    • not much value-add
Update Testing (Henrik)
  • Currently we fail in downloading partial updates because the about window and the old ui are blocking each other. (bug 599290)
    • Attached is a workaround patch
  • We will have to update the shared module asap to be able to test the new ui
Add-ons Testing (Henrik)
  • With Mozmill 1.5.1 add-on authors should be able to create tests now. But should we wait with advertisement until shared modules have been refactored?
  • Dave is working on tests for the Selenium IDE extension
Mozmill Crowd Extension (Henrik)
  • In contact with Patrick Brunswig regarding IPC-PIPE regression. Will still use nsIProcess to prevent possible other upcoming regressions
Mozmill Result Dashboard (Henrik)
  • Waiting to land patches for software update tests (multiple updates)
Automation / Infrastructure (Henrik)
  • Investigating possible notification exchange between RelEng/QA for triggered test-runs

Personal Status

For the personal status please check the weekly status updates:

Roundtable

General
  • Results from work week
  • Endurance tests with Mozmill (see Etherpad document)
  • MozMill testday on Nov 26th?
    • Yes, we want to have it
  • Litmus vs. Mozmill tests
    • For the add-ons manager we have 60+ in-litmus requests. Do we want to have manual tests for all those requests or can we only implement those which really need manual action?
  • Other
    • Etherpad
      • Any comments? What do you like? What don't you like? Any suggestions?
        • I can directly see the status of any team member without having to look at their status reports (Henrik)
        • I can see discussions even when I was offline (Henrik)
        • Etherpad makes it so easy and fast to edit content compared to the wiki (Henrik)
Issues

Meeting Notes

  • Goals Review and Additions
    • [CARRY OVER] Mozmill Crowd extension
      • Test environments for Windows and Linux have to be setup
      • Need functionality to download and install the test environment
      • Update the extension to use nsIProcess instead of IPC-PIPE component
      • Demo of the extension during the all hands week
    • [ON TRACK] Mozmill Dashboard (update and add-on tests)
      • Once the automation script sends all the data up to brasstacks (later this week) we have to think about the visualization of the results
      • For add-on tests we will setup an addon testrun for Selenium IDE
    • [ON TRACK] Buildbot execution of tests with local test data
      • behind on creating new local test files - we can use the test day to get progress
      • 35 tests moved to local test data last quarter
      • 4 test files for this quarter / about 20 test files remaining
      • we should make sure that tests with already local test data are stable and can go to buildbot
      • what is the plan to get a specific set of tests to get run on buildbot
        • tag tests with meta data in the file which can be used by a script to copy over
    • [ON TRACK] Refactoring the shared module system
      • not a lot of progress in the work week
      • implementation of the initial classes
      • we will have at least the majority pieces in place to implement most of our tests
      • pick tests from nearly each of the existing test groups for a broad coverage
    • [ON TRACK] Mozmill Panorama tests
      • Lead needed for coordination -> Cameron? Who is 2nd?
      • Geo will talk with Clint about drag and drop requirements
      • Given by Anthony there should be 1/2 of the tests on Litmus be able to run without D&D
      • Geo will come up with some performance tests
      • Focus on already existent tests on Litmus
        • Figuring out with QA leads and devs which of those tests are important to automate
    • [NEW] Prototype for endurance tests
      • For first thoughts see the etherpad
      • Have to find a lead and 2nd
    • [NEW] Define trigger mechanism to start Mozmill test-runs
      • whatever sort of notifications even manual ones to start test-runs on all the platforms
      • Send out emails when failures happen
      • Big impact of Mozmill tests to run RelEng update tests
      • Talk to RelEng to define the type of notification (Lead: Geo, 2nd: Henrik)
        • No ETA for RelEng inclusion
        • We have requested a way to get notification from RelEng for our test-runs
        • Make our update tests absolutely trustful before running them in the RelEng environment (i.e nightly-test channel)
  • Handling of manual testcases in Litmus for automated tests
    • Each automated test needs a manual test
    • Where they have to be located?
      • Keep in mind that Litmus 2 is not that far away.
      • jsdoc vs attachment on Bugzilla
        • jsdoc can handle changes more gracefully - one step with script changes
        • jsdoc comments can be automatically pulled
      • Everything should be on one place, mark tests for automation only

Action items

  •  ? Al (Anthony)?: Check existing tests if they are already covered by mochitests and worth to transform local
  • Aaron (Geo): Tagging of tests and how to get them into buildbot
  • [DONE] Geo (Henrik): Schedule meeting about trigger notifications for test-runs
  • [DONE] Geo: Start thread about which parts the Etherpad document should cover
  • [DONE] Matt (Henrik): Finding a lead and 2nd lead for the Panorama goal
  • [DONE] Matt (Henrik): Goals review and addition of new goals to team goals
  • [ON TRACK] Henrik: Work with Waverly on updating the spreadsheet for Add-ons Manager tests - add labels for automated tests