QA/Test Automation/2010-11-17
From MozillaWiki
< QA | Test Automation
previous meeting | Meetings | next meeting »
Contents
Dial in
# 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL) # 1-800-707-2533 (pin 369) Conf# 315 (US) # irc.mozilla.org #mozmill for backchannel
Attendees
- Cameron, Geo, Al, Matt, Anthony, Aaron, Adrian, Henrik, Dave
Last weeks action Items
Goals Overview
Risky Goals
Project Status
- General (Henrik)
- Waiting for Clint's proposal how to do Drag&Drop - needed for Panorama tests
- Broken Tests
- Firefox 4.0 (Geo)
- Firefox 3.5/3.6 (Anthony)
- Landed waitForTabPanel() to make notification bar handling more reliable (5 of top-10 fails resolved)
- DOM Walker + L10n API (Adrian)
- final review stages
- Redesign Shared Modules (Geo)
- Test Refactoring (Anthony)
- Draft styleguide being composed (targetting EOW)
- Begin refactoring once styleguide approved
- Local Test Data (Anthony)
- No progress yet - considering community outreach to create testpages
- 35 tests are already local
- 4 tests remain which can be made local
- remainder are not doable in Mozmill
- not much value-add
- Update Testing (Henrik)
- Currently we fail in downloading partial updates because the about window and the old ui are blocking each other. (bug 599290)
- Attached is a workaround patch
- We will have to update the shared module asap to be able to test the new ui
- Add-ons Testing (Henrik)
- With Mozmill 1.5.1 add-on authors should be able to create tests now. But should we wait with advertisement until shared modules have been refactored?
- Dave is working on tests for the Selenium IDE extension
- Mozmill Crowd Extension (Henrik)
- In contact with Patrick Brunswig regarding IPC-PIPE regression. Will still use nsIProcess to prevent possible other upcoming regressions
- Mozmill Result Dashboard (Henrik)
- Waiting to land patches for software update tests (multiple updates)
- Automation / Infrastructure (Henrik)
- Investigating possible notification exchange between RelEng/QA for triggered test-runs
Personal Status
For the personal status please check the weekly status updates:
Roundtable
- General
- Results from work week
- Endurance tests with Mozmill (see Etherpad document)
- MozMill testday on Nov 26th?
- Yes, we want to have it
- Litmus vs. Mozmill tests
- For the add-ons manager we have 60+ in-litmus requests. Do we want to have manual tests for all those requests or can we only implement those which really need manual action?
- Other
- Etherpad
- Any comments? What do you like? What don't you like? Any suggestions?
- I can directly see the status of any team member without having to look at their status reports (Henrik)
- I can see discussions even when I was offline (Henrik)
- Etherpad makes it so easy and fast to edit content compared to the wiki (Henrik)
- Any comments? What do you like? What don't you like? Any suggestions?
- Etherpad
- Issues
Meeting Notes
- Goals Review and Additions
- [CARRY OVER] Mozmill Crowd extension
- Test environments for Windows and Linux have to be setup
- Need functionality to download and install the test environment
- Update the extension to use nsIProcess instead of IPC-PIPE component
- Demo of the extension during the all hands week
- [CARRY OVER] Mozmill Crowd extension
- [ON TRACK] Mozmill Dashboard (update and add-on tests)
- Once the automation script sends all the data up to brasstacks (later this week) we have to think about the visualization of the results
- For add-on tests we will setup an addon testrun for Selenium IDE
- [ON TRACK] Mozmill Dashboard (update and add-on tests)
- [ON TRACK] Buildbot execution of tests with local test data
- behind on creating new local test files - we can use the test day to get progress
- 35 tests moved to local test data last quarter
- 4 test files for this quarter / about 20 test files remaining
- we should make sure that tests with already local test data are stable and can go to buildbot
- what is the plan to get a specific set of tests to get run on buildbot
- tag tests with meta data in the file which can be used by a script to copy over
- [ON TRACK] Buildbot execution of tests with local test data
- [ON TRACK] Refactoring the shared module system
- not a lot of progress in the work week
- implementation of the initial classes
- we will have at least the majority pieces in place to implement most of our tests
- pick tests from nearly each of the existing test groups for a broad coverage
- [ON TRACK] Refactoring the shared module system
- [ON TRACK] Mozmill Panorama tests
- Lead needed for coordination -> Cameron? Who is 2nd?
- Geo will talk with Clint about drag and drop requirements
- Given by Anthony there should be 1/2 of the tests on Litmus be able to run without D&D
- Geo will come up with some performance tests
- Focus on already existent tests on Litmus
- Figuring out with QA leads and devs which of those tests are important to automate
- [ON TRACK] Mozmill Panorama tests
- [NEW] Prototype for endurance tests
- For first thoughts see the etherpad
- Have to find a lead and 2nd
- [NEW] Prototype for endurance tests
- [NEW] Define trigger mechanism to start Mozmill test-runs
- whatever sort of notifications even manual ones to start test-runs on all the platforms
- Send out emails when failures happen
- Big impact of Mozmill tests to run RelEng update tests
- Talk to RelEng to define the type of notification (Lead: Geo, 2nd: Henrik)
- No ETA for RelEng inclusion
- We have requested a way to get notification from RelEng for our test-runs
- Make our update tests absolutely trustful before running them in the RelEng environment (i.e nightly-test channel)
- [NEW] Define trigger mechanism to start Mozmill test-runs
- Handling of manual testcases in Litmus for automated tests
- Each automated test needs a manual test
- Where they have to be located?
- Keep in mind that Litmus 2 is not that far away.
- jsdoc vs attachment on Bugzilla
- jsdoc can handle changes more gracefully - one step with script changes
- jsdoc comments can be automatically pulled
- Everything should be on one place, mark tests for automation only
Action items
- ? Al (Anthony)?: Check existing tests if they are already covered by mochitests and worth to transform local
- Aaron (Geo): Tagging of tests and how to get them into buildbot
- [DONE] Geo (Henrik): Schedule meeting about trigger notifications for test-runs
- [DONE] Geo: Start thread about which parts the Etherpad document should cover
- [DONE] Matt (Henrik): Finding a lead and 2nd lead for the Panorama goal
- [DONE] Matt (Henrik): Goals review and addition of new goals to team goals
- [ON TRACK] Henrik: Work with Waverly on updating the spreadsheet for Add-ons Manager tests - add labels for automated tests