QA/Test Automation/2010-12-15
From MozillaWiki
< QA | Test Automation
previous meeting | Meetings | next meeting »
Contents
Dial in
# 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL) # 1-800-707-2533 (pin 369) Conf# 315 (US) # irc.mozilla.org #mozmill for backchannel
Attendees
- Aaron, Antony, Al, Henrik, Geo, Dave, Cameron
Last weeks action Items
- [DONE] Matt/Henrik: final leads for Panorama
- [DONE] Matt/Henrik: final priority list/leads for open project
- [DONE] Dave/Aaron: consolidate past discussion re: Endurance tests into one Etherpad
- [DONE] Dave/Aaron: project page for Endurance Tests
- [ON TRACK] Anthony: (Panorama) project page/tracking bugs/clarification of which tests to implement
- [DONE] Adrian: update L10N section of notes
Goals Overview
Risky Goals
Project Status
Endurance Tests (Dave/Aaron)
- Project wiki page updated with consolidated notes taken during yesterdays meeting
- Roadmap written with Q4 goal plan
Panorama Tests (Anthony/Al)
- Currently investigating existing tests for potential overlaps in testing
Triggered software updates (Geo/Henrik)
- Triggered Update Tester wiki page
- We are going with Pulse
- Have all pieces in place except the download code (am assured it's very simple)
- Are gluing pieces together today, targeting demo tomorrow
Mozmill Crowd Extension (Henrik/Aaron)
- Moved spawning external processes to nsIProcess
- Implemented downloader for test environment
- More stripped down version of the Windows test-environment; Linux and OS X under construction
- Talked with Jorge about a place to host test environments - HTTPS via QMO?
- Shared Modules Refactor wiki page
- Have made lots of progress getting basic Element/Widget binding going
- Frequent code updates happening right now while map structure is defined
- See Geo's github repo for more details
Mozmill Result Dashboard (Henrik/Anthony)
- Mockups for update test results
- Investigating Elastic Search for CouchDB replacement
Build bot - local data (Anthony/Aaron)
- nothing new to report
Others
- DOM Walker + L10n API (Adrian)
- Short updates
- Tests / Shared Modules
- General (Henrik)
- TabView shared module has been updated, there shouldn't be any dependencies anymore
- Readme files in repository have been populated with content
- Test Refactoring (Anthony)
- Phase 1 work defined
- Need final say on use of "==" vs "===" in conditionals
- Broken Tests
- Firefox 4.0 (Geo)
- No progress
- Firefox 3.5/3.6 (Anthony)
- nothing new to report
- Firefox 4.0 (Geo)
- Automation Scripts / Infrastructure
- Short updates
Personal Status
For the personal status please check the weekly status updates:
Roundtable
- General
- Issues
- How we handle broken tests in order to make test runs and litmus syncs cleaner
- We should fix broken tests as soon as possible, but history tells us they stick around for awhile so we need a process to handle them while in progress
- Even under perfect circumstances, some tests are hard to fix. This process will be needed for those if no others.
- Proposed flow
- Check to see if it has failed (twice?) during the last seven days, if so:
- Open a bug on the test failure
- Add whiteboard tag to the bug for tracking current failures
- Remove test from mozmill Litmus test suite (make sure it is still in BFT and/or Smoketest manual suites)
- Mark it for Mozmill as a skip test with the bug number from #2.
- Once fixed,
- Unmark skip
- Replace in Litmus
- Remove whiteboard tag
- Resolve bug
- Check to see if it has failed (twice?) during the last seven days, if so:
- Proposed flow
Meeting Notes
- Endurance Tests
- It's not realistic to have a wrapper script by end of this year (personal constraints)
- For now don't use a Python wrapper script, but a faked JSON object in the test itself to populate the data
- Need to find more ideas what the exact next steps will be
- We need more personal resources for coding Python
- Panorama
- Work together with panorama devs
- Aza is leaving, so Ian Gilman is our contact person now
- Identifying extisting Mochitests one by one to find most useful Mozmill tests
- Geo still has some performance stuff in petto but no time to complete it
- Could be part of the endurance tests
- Performance tests with checkpoints in the background (dependencies on Mozmill)
- would be great to have 5 tests for the top nasty bugs by end of the year
- Get list of high priority tests to get written
- Most likely some restart tests
- Work together with panorama devs
- Triggered Updates
- With the work from today a demo should be possible tomorrow
- Create a screencast for the demo
- With the work from today a demo should be possible tomorrow
- Mozmill Crowd
- Demo tomorrow during the QA Roadmap meeting
- Refactor Modules
- High hierarchy map could be auto-generated by Mozmill
- Silktest style for the map management, selenium style for page objects
- Dashboard
- Sanity check for number of tests run - 0 passed tests should be treated as fail bug 620386
- Matrix for release candidate testing next year
- Elastic search
- Get use cases so we can make sure it fits all our expectation
- Regression Test
- What's the process how to create mozmill tests from bugzilla
- Heathers regression tool (bisect)
- Others
- Coding Style: Always use === in our tests
- How to treat test failures
- Add litmus test id into bug
- Always disable tests, even when they only fail on one single platform
- Link from mozmill tests to litmus tests
- We really shouldn't add id's to tests because of Litmus branches (Litmus 2?)
- Should be easier via a Litmus search query
Action items
- Geo: Update documentation for handling of failing Mozmill tests
- Anthony: Update coding styles and keep discussion on-going to get remaining important rules defined