QA/Test Automation/2011-01-26
From MozillaWiki
< QA | Test Automation
previous meeting | Meetings | next meeting »
Contents
- 1 Dial in
- 2 Attendees
- 3 Last weeks action Items
- 4 Goals Overview
- 5 Risky Goals
- 6 Project Status
- 7 Personal Status
- 8 Roundtable
- 9 Meeting Notes
Dial in
# 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL) # 1-800-707-2533 (pin 369) Conf# 315 (US) # irc.mozilla.org #mozmill for backchannel
Attendees
- Anthony, Dave, Aaron, Al, Cameron, Geo, Matt, Henrik
Last weeks action Items
- [MISSED] Geo: Update documentation for handling of failing Mozmill tests
- At this point, have lost the thread on this. Need to touch base with Henrik to find out what final plan was and where docs should be updated.
- [DONE] Anthony: Update coding styles and keep discussion on-going to get remaining important rules defined
Goals Overview
Risky Goals
Project Status
On-demand update (bft) tests (Geo/Henrik)
- Project page started here, still filling in info.
- Henrik has made great progress on staging downloads
- Other part is simultaneous test launch
- Currently planning on using ssh/dsh, or similar
- Will be staging a test version on qa-mozmill
Endurance Tests (Dave/Anthony)
- Tracking bug created. bug 629065
- Patch uploaded for shared-module and initial test snippet. bug 629069
- Patch uploaded for endurance testrunner. bug 629077
- Created branch for endurance tests reports in Mozmill dashboard. bug 629083 repository
- Reports have been generated using the current patches and published to a couchone account:
Panorama Tests (Anthony/Al)
- Community contributor (uyclay) has 4 tests which are nearly ready to checkin
- Drag&Drop issue bug 515776 "resolved"
- All tests should now be non-blocked, in theory
- If you want to help out, see the list and assign something to yourself
- Tentative project plan has been added here. Still undergoing review b/w Henrik and Geo, but will be finalized this week.
- Will be sending out periodic information for review by team as we hit milestones
- Plan on team-wide pilot at the end of Q1, where everyone can try implementing tests against the library and give feedback.
- Waverley has also expressed interest in ramping up into automation development; they can help us out a lot by being a "test community."
Shadow test server (Al/?)
- Initial Apache server being created in an IT-run ESX server. See bug 621794.
- Once server is up, will need to create some sort of rsync script to mirror content from QA repo initially.
- Will also need to determine what certificates we want for https testing.
Others
Mozmill Crowd
- Mozmill Crowd 0.1 and an improved version 0.1.1 has been released
- We already have some daily users (81 downloads, 8 active users)
- Working on project page updates and milestones for this quarter
- Next planned version of Mozmill Crowd by next week
- Mozmill Crowd talk during FOSDEM 2011 on Sunday, Feb 6th
Dashboard
- Working on project updates and milestones for this quarter
- Implementation of necessary views for l10n tests earlier this week
Cloud Automation (Al/?)
- Creating initial project page
- Determining initial milestones, requirements, and dependencies.
- This is investigation in the immediate, near-term
- General (Henrik)
- No progress - first Panorama tests still under review
- Broken Tests
- Firefox 4.0
- Landed a lot of fixes; down to one single test failure on Linux
- Firefox 3.5/3.6 (Anthony/Aaron)
- Bug with window.close() not being properly handled (bug 626674) - fix landed, will check testrun tomorrow to check for residual failures resolved
- Bug with testPasteLocationBar (bug 624682) - reverted from new controller MenuAPI to using the old controller click access of popup menu items, revealed another issue on 3.5 with keypress copying
Automation Scripts / Infrastructure (Henrik)
- Download script for builds has been finished and can be used
- Work happens now on the wrapper script to auto-start download and the test-run.
Personal Status
For the personal status please check the weekly status updates:
Roundtable
- General
- Testing of 32bit builds on Ubuntu 64?
- Value of the execution of Mozmill tests for all locales (Henrik)
- l10: A huge amount of issues has been covered by the testrun
- Results
- Screenshots
- Tests take about 80 minutes
- Update: 1 crasher bug has been identified for 3 locales (bug 628829)
- Results
- Tests take about 2h for partial updates
- Pros vs. Cons
- Check stability and quality of all localized builds
- No need for manual update spot checks
- Limited number of machines (what about the new order?)
- We should test from Tier1 -> Tier3 (limit to Tier1 for smoketests and test channels?)
- What about BFT tests? A single test-run for a locale will take about 12 minutes. We could run in parallel while Waverly tests manually
- l10: A huge amount of issues has been covered by the testrun
- Issues
Meeting Notes
- Endurance
- A valuable resource of information would be graphs over graphs
- Compare test-runs to see improvement/degradation of memory usage or performance
- Shadow Test Server
- We need a 2nd lead on the project.
- Send an email to the list if you have interest
- Otherwise Henrik will take the position
- Mozmill Dashboard
- Overlay results with appropriate bug ids
- Overlay results with additionally meta data added by the logged in user
- General
- Everyone should check the own handling of projects and collect ideas for improvements in next quarter
- For test failures we have to focus on 3.6 test failures before 3.5 failures
Action items
- Anthony
- Update refactoring discussion items and communicate to team
- Make sure spreadsheet is updated with Panorama status
- Al
- Talk with Anthony to find the requirements for SSL certificats on the shadow server
- Geo
- Finish on-demand updates project page
- Figure out what jetpack is doing for docgen (ask myk/analyze sdk)
- Document failed test process from henrik's email at appropriate place
- Henrik
- Send Geo desired place for failed test docs
- Add 32-bit version of Firefox to our daily testruns on Ubuntu 64-bit
- Matt/Al
- Matt/Al: work to find the goal for cloud automation goal