B2G/QA/Meetings/Automation/2014-12-11

Overview

This is a weekly meetup to discuss all things regarding B2G automation and Quality. Please use wiki style.

• Meeting Name: QA Automation Roundtable
• Meeting Time: Thursday @ 9:30am - 10am PST
• Vidyo Room: QA Vidyo Room
• Video Link: https://v.mozilla.com/flex.html?roomdirect.html&key=nL4PESn1IaVh
• Dial in: 1-800-707-25331
• Meeting Notes: https://wiki.mozilla.org/B2G/QA/Meetings
• Previous Meeting:https://wiki.mozilla.org/B2G/QA/Meetings/Automation/2014-11-27
  • Q4 Goals B2G (Draft)
    • Complete 2.1 QA criteria: https://wiki.mozilla.org/Release_Management/FirefoxOS/Release_Milestones#Code_Complete_Criteria
    • Identify 3-5 active contributors that has completed One and Done tasks
    • Identify 2.2 features and have testplans drafted by end of sprint 2
    • Automate 20 gaia UI tests from backlog. have them running on device [geo: add "if appropriate"?], and checked into the main repo. and reporting against Tree Herder
      • [geo] Running on device (for MarionetteJS) and reporting to Treeherder are both harness features and belong to FxOS Auto, delivery of those is out of scope for QA. We should stick to "automate 20 gaia UI tests," implication being against whatever harness is currently supported and reasonably stable. I'd suggest leaving out "from backlog"--that's an implementation detail.
    • Clean up 2.1 invalidated testcases and prepare for 2.2 testsuite
      • performance goal for acceptance testing

Attendees

• Naoki
• No-Jun
• Otilia
• Peter Bylenga
• Johan
• Vio
• Bebe
• Robert
• Geo
• Martijn
• Oliver Nelson

Previous Action Items

• [DROPPED] [Tony] Hold off on adding extra SIMs for now, and only keep it on b2g-22.1 for testing. This device has been isolated from the grid from the general pool of tests. Tony will also look into expanding if we can add other flames for dsds coverage
• [CARRY OVER] [Tony] Talk to TwQA today about reaching out for 2.1 on device smoketests
  • Softvision is waiting for a go to stop the automation on 2.1. you can stop.
  • New tests on master
  • Work in progress, uplift to 2.1 as a judgment call (verify with Tony)

Status Highlights

• 2.2 is a go (master)

Agenda

• [Tony] how can we improve and accelerate the Review process? is there a common best practices page for tests? I'm hearing that it takes longer to review a test than to write one.
• [Otilia] SoftVision PTO planning around holidays - you can find the dates here https://docs.google.com/spreadsheets/d/1cWHkm2SAoAG0TlLJK-h-QEeQyfyTKDID2ww5YtxcOPo/edit#gid=0
• US/EU PTO:
  • Geo: 12/22-01/02
  • Johan: 12/24-12/28 & 01/01

US/EU Holidays: Christmas Eve - December 24 Christmas Day - December 25 New Year's Day - January 1

This Week - Decisions & Action Items

• [Geo] best practices page for reviews
  • PEP8
  • WebQA style guide
  • Overall hg/Mozilla standards
  • Correctness
  • Clarity
  • Won't pass by mistake, fails loudly
  • How to do commit messages, squashes, PRs, etc.
  • Putting initial testing in patch comments
  • When to run locally on review

12/18 agenda

• [jlorenzo] Creating another run for less important tests (like https://bugzilla.mozilla.org/show_bug.cgi?id=1083731 )