previous meeting | Meetings | next meeting »

Dial in

 # 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL)
 # 1-800-707-2533 (pin 369) Conf# 315 (US)
 # irc.mozilla.org #mozmill for backchannel

Attendees

• Al
• Geo
• Cameron
• Aaron
• Anthony
• Henrik
• Dave Hunt

Last weeks action Items

•  ? Al (Anthony)?: Check existing Panorama tests if they are already covered by mochitests and worth to transform local
  • No progress
• Aaron (Geo): Tagging of tests and how to get them into buildbot
  • Can do a tag recognition script in about an hour or two if we're willing to accept bash+grep
  • Translating it into something more portable like Python will take extra time
  • Suggest letting me do the bash+grep version initially, then following up w/ Python
  • Open question: manifests or in-test tagging? Need to figure out ETA/effort estimate on manifest solution
• [DONE] Geo (Henrik): Schedule meeting about trigger notifications for test-runs
• [DONE] Geo: Start thread about which parts the Etherpad document should cover
  • Consensus is towards:
    • Highlights from week
    • General section for team-general stuff
    • Personal status
• [DONE] Matt (Henrik): Finding a lead and 2nd lead for the Panorama goal
  • Need final confirmation from Matt
• [DONE] Matt (Henrik): Goals review and addition of new goals to team goals
  • Need final priorities, etc. from Matt
• [ON TRACK] Henrik: Work with Waverly on updating the spreadsheet for Add-ons Manager tests - add labels for automated tests
  • Ongoing task

Goals Overview

• QA Q4 goals

Risky Goals

Project Status

Endurance Tests (Aaron)

• Lead: Aaron Train
• Second: Dave Hunt
• Team Assistance: Geo Mealer, Henrik Skupin

• • Hoping to kick this off today - Dave/Aaron
  • Kickoff ideas/brainstorming - notes

Panorama Tests (Anthony)

• Lead: Anthony Hughes
• Second: Al Billings
• Team Assistance: Aaron Train, Dave Hunt

• 15 / 21 identified tests do not rely on Drag & Drop
• ashughes:
  • Spoke with Matt about taking lead on this and I am interested
  • I know Al and Cameron were interested as well so can we get a final word on this?

Triggered software updates (Geo)

• Lead: Geo Mealer
• Second: Henrik Skupin
• Team Assistance: Anthony Hughes

• Note: If not already done by meeting, will be moving this into a project page.
• Proposed structure:
  1. Trigger comes from somewhere on build landing, contains current build id
     • Open question: Pulse vs. Email from Releng? In discussions now on both sides.
  2. Trigger polling script on VM catches trigger
  3. Trigger polling script adds build to queue (directory full of semaphores or a flat file) for processing.
     • May not be needed for initial PoC as builds don't typically come frequently enough to queue. This would be an edge case handler.
  4. Queue polling script pulls the job off the queue, does two things:
     1. Downloads current build and saves it off for next trigger.
     2. Build saved off from last trigger will be updated and compared to expected build id.
  • Initial reporting to brasstacks, possiblly email on failures
  • Initial version for trunk, en_US only. Still targeting multi-platform.
• Things already there
  • Script to update a build and do some basic heuristics for testing exists.
• Things needed now
  • Update test script needs to be able to take a particular build id for verification
  • Trigger being decided upon between QA, RelEng, and Christian Legnitto (Pulse)
    • Eventually we want to be on Pulse, is unknown now whether it's sufficiently mature.
    • Go/no-go on Pulse to be made no later than next week, possibly as early as Friday.
    • Overall structure the same regardless.
  • Polling script
  • Queuing mechanism (? see above)
• Things needed eventually
  • RelEng wants to do this in their infrastructure eventually if at all possible.
  • If not possible, we'll need a way to report directly back to RelEng
  • Not there yet, though. For now want to run it on our side to make sure of robustness before putting in critical path.
• Complications
  • If we can't get per-platform notifications, may have to add what amounts to a multiplexer
    • Multiplexer would take one final notification and then notify all the VMs.
    • I'd rather not have to do this (it's complicated, delays testing, and possibly introduces deadlocks) so am seeking per-platform notifications.

Mozmill Crowd Extension (Henrik)

• Lead: Henrik Skupin
• Second: Aaron Train

• Currently: Collecting dependencies and building test-environments for all platforms
• Next: move to nsIProcess API

Redesign Shared Modules (Geo)

• Lead: Geo Mealer
• Second: Henrik Skupin
• Team Assistance: QA Automation Team

• Slow progress to date largely due to priority juggling
• Plan of Action still the same:
  • Implement 80% case controls (button, textbox, menus/lists)
  • Implement control constructors that adapt DOMUtils and Elementlib methods of control binding
  • Implement 80% case methods (click, type, focus control, text verification)
  • Implement a sample test from as many subdirectories as possible for breadth.
  • 20% case controls/methods/etc to follow, but seeking max value up front.
• Class declaration is straightforward
• Map declaration less straightforward, attempts to date have been a little messy
• Timeboxing map declaration decision to next week
• After decision made, actual creation of map and conversion of tests is very quick.

Mozmill Result Dashboard (Henrik)

• Lead: Henrik Skupin
• Second: Anthony Hughes

• Reworking project pages
• Next: Proposal for visualization and queries of update results

Build bot - local data (Anthony)

• Lead: Anthony Hughes
• Second: Aaron Train

• No change - high resource cost, low value at this point
• Doable: 4 test pages + 1 test
• Much depends on proxy and certificate server handling which is not currently doable
• Not likely to see any change this quarter

Others

DOM Walker + L10n API (Adrian)

• Pushed to 2.0 and 1.9.2 branch (including the Preferences-window tests); 1.9.1 seems to be not easy to port - investigating that further
• some follow-up bugs left. I plan to work on them during this month.

Tests / Shared Modules

• General (Henrik)
  • Refactored modal dialog has been landed on all branches
    • Some hidden older failures have been revealed
    • Some regressions introduced
    • Interests from Simon Steward to use the module for Selenium

• Test Refactoring (Anthony)
  • Styleguide drafted - final call for review/feedback
    NOTE: Please be aware that there are some things left out of this version of the styleguide. The initial version serves to communicate those things we have all already agreed on. Discussion and revision will improve on version 1.0.
    • Matt: What is the criteria of review comment vs review rejection? We should be very explicit on this so that there are no surprises with review results. NOTE: This might be more appropriate for this document and linking the two.
  • Tasks for Phase I have been outline and will go into a project page shortly - expect work to begin next week

• Broken Tests
  • Firefox 4.0 (Geo)
    • Dave Hunt also working on this
    • bug 592411 fixed, which fixed several topfails
    • Still continuing to pick tests off the Top Failures List and fixing as opportunity/time arises
  • Firefox 3.5/3.6 (Anthony)
    • 3.6: 18 Failures - up from 13 since modalDialogAPI
    • 3.5: 18 Failures - up from 9 since modalDialogAPI
    • Aaron focusing on modalDialogAPI refactoring in tests
    • Anthony focusing on pre-existing failures
    • Fix Rate: 3/week

Automation / Infrastructure

• Update Testing (Henrik)
  • Reports now support multiple updates in a row
• Add-ons Testing (Henrik)
  • no update

Personal Status

For the personal status please check the weekly status updates:

• 101129 - 101203

Roundtable

General

• Project management software (Henrik)
  • See weekly etherpad
  • Proposal for wiki pages

Issues

Meeting Notes

• Notes on last week's action items under items themselves
• Endurance Tests
  • Dave/Aaron picking up, have basic project management stuff to do
• Panorama tests
  • Anthony provisionally picking this up (need final confirmation), again project management stuff to-do
• Update Testing
  • Notes inline
  • PoC/demo targeted for all-hands
• Crowd Extension
  • Notes inline
• Refactor
  • Notes inline
  • Targeting competing map examples by all-hands so we can make a team decision
• Dashboard
  • Notes inline
  • Also working on direct linkification into dashboard
• Buildbot/Local Test data
  • Notes inline, no changes expected
• L10N
  • Notes inline
• Tests/Shared Mods (General)
  • Notes inline
• Test Refactoring (Style Guide)
  • Notes inline
  • Style Guide almost final, Anthony waiting for last bits of feedback
  • r- criteria question still open. Proposed to wait for all-hands? Need Matt's feedback
• Broken Tests
  • Notes inline
  • Be careful of big checkins like Modal Dialogs. Make sure to take into account for triage.
  • Can sort by multiple fields using shift-click in couchapp
  • Branch failures can inform us for trunk failures too, let's make sure to keep communication clear
• Automation/Infrastructure
  • Notes inline
  • Even though reports support multiple updates in a row, the Mozmill harness doesn't. Mozmill needs light changes to fully support process.
  • Multiple updates proposed for 1.5.2, may bump to 1.5.3.
• Round Table
  • Project Management Tools
    • Notes inline
    • Mention leads directly on wiki page w/ link to project pages
    • Use existing tools
    • Maybe use bugzilla more thoroughly with whiteboarding
    • tabled until all-hands, Henrik to give ideas of what he has in mind w/ whiteboarding

Action items

• Matt/Henrik: final leads for Panorama
• Matt/Henrik: final priority list/leads for open project
• Dave/Aaron: consolidate past discussion re: Endurance tests into one Etherpad
• Dave/Aaron: project page for Endurance Tests
• Anthony: (Panorama) project page/tracking bugs/clarification of which tests to implement
• Adrian: update L10N section of notes