QA/Test Automation/2011-01-12
From MozillaWiki
< QA | Test Automation
previous meeting | Meetings | next meeting »
Contents
Dial in
# 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL) # 1-800-707-2533 (pin 369) Conf# 315 (US) # irc.mozilla.org #mozmill for backchannel
Attendees
- Geo, Matt, Al, Henrik, Aaron, Anthony, Adrian
Last weeks action Items
- Geo: Update documentation for handling of failing Mozmill tests
- Did not update this, but the handling is somewhat up in the air right now. After the roundtable from this meeting should know better how to proceed.
- [DONE] Anthony: Update coding styles and keep discussion on-going to get remaining important rules defined
Goals Overview
- Proposals
- Triggered Update Tests
- Integrate triggered Mozmill update tests into nightly automation process
- One goal, but lower priority because it's an incremental change
- what about release candidate builds? For Q2?
- Release candidate tests are the higher priority for Q1. They will be on-demand, rather than triggered
- Separate goal
- Measurement will be "Matt can run them."
- Integrate triggered Mozmill update tests into nightly automation process
- Endurance Tests
- Create suite of Mozmill tests to stress-test Firefox in different areas
- Performance tests
- Module Refactoring
- Finalize the project by updating modules and tests
- Only for mostly used modules and tests for the beginning (focus areas)
- Updates per component
- Finalize the project by updating modules and tests
- Collaborate with development teams to get XXX BFT tests automated
- We never found a good way to specify a realistic goal, whether via number of tests nor per component. Do we have other options?
- Given the Firefox 4 priority and other goals, how much time we would have this quarter?
- Note: let's carry over panorama test goal for this.
- Should we better move our efforts to keep the testsuite green?
- Not a Collaboration goal, can be a process goal
- Testing Server
- Setting up a testing server which allows kill our dependencies on external websites.
- iterative effort
- need to create a feature milestone implementation plan to define what will be delivered in chronological order
- Setting up a testing server which allows kill our dependencies on external websites.
- Triggered Update Tests
Risky Goals
Project Status
Panorama Tests (Anthony/Al)
- 4 / 15 doable tests currently in development/review
- 6 tests currently blocked by drag & drop (bug 515776)
- Will be looking for team assistance once 4.0b9 work dies down (early next week)
- Proposed Team Goal: All non-blocking tests automated
Software updates (Geo/Henrik)
- Finished PoC for triggered nightly updates.
- Takes Pulse message
- Downloads appropriate build
- Runs update tests, including checking that end-build == expected build ID.
- Emphasis on updates for next few weeks will be streamlining on-demand release updates, since that's expected to be the immediate process pain point
Mozmill Crowd Extension (Henrik/Aaron)
- Version 0.1 is live on AMO
- We have to inform localizers ASAP so they can run our tests against their locales and fix remaining issues before RC1
- Version 0.2 planned for end of this month (non-blocking process)
- Submitted proposal for my talk at FOSDEM 2011
Endurance Tests (Dave/Aaron)
- Update project page with notes from this week's meeting, roadmap redefined
- PoC stage reached in December, running "toy" tests.
- Currently porting real tests over to shake out early-stage issues.
- Determining a milestone-based plan including pilot periods where the team as a whole can help contribute to porting (and also test out how easy the framework is to develop against)
Mozmill Result Dashboard (Henrik/Anthony)
- Update tests now have their own views
- A couple of minor issues identified which have to be fixed
Build bot - local data (Anthony/Aaron)
- No status
- Most likely will not see progress this quarter due to Firefox 4 work and Automation priorities
Others
- DOM Walker + L10n API (Adrian)
- Slower progress but working on the last major items to get fixed by Adrian
- For the most important bug left (bug 614579), we need input from people knowing about theming stuff, as it seems to be a Firefox bug...
- Can be executed by the Crowd extension
- Tests / Shared Modules
- General (Henrik)
- Nothing major to report
- Test Refactoring (Anthony)
- Following is moot until Shared Modules are refactored
- project in place to begin once Firefox 4b9 work dies down
- will need team assistance to complete everything this quarter (please let me know if you are willing to help)
- we still have a few outstanding items for discussion
- if possible, I'd like to discuss these during the team meetings (resolve 1 per meeting)
- agreement would be integrated into refactoring, not styleguide
- Broken Tests
- General
- We have started combined efforts to get some of the failing tests fixed
- We have now owners for the different sections which have to take care of broken tests
- Firefox 4.0 (Geo)
- No particular branch-based news, see General above and Round Table below.
- Firefox 3.5/3.6 (Anthony)
- Focusing on 4.0 failures for b9 release
- Will focus on branch failures once b9 is out
- General
- Automation Scripts / Infrastructure
- Download script for builds from FTP is in progress. It already works for release and candidate builds
Personal Status
For the personal status please check the weekly status updates:
Roundtable
- General
-
Styleguide & Refactoring- on hold until Shared Module refactoring complete- we should fork into two separate projects
- freeze the styleguide
- new additions go into refactoring work
- changes/additions to styleguide should reflect "style" not logical/design changes
- Failures
- adopt a module-ownership model to deal with all failures (4.0, 3.6, and 3.5)
- once we are green, go back to a branch-based model or stay with module-ownership?
- Issues