B2G/QA/2.2/2015-03-02Status

V2.2 Highlight & Risk

• Highlight
  • Entering V2.2 FL
  • Current running tasks
    • Flame-kk manual smoke test
    • Flame-kk automation test
    • Flame-kk MTBF - less than 10 hours.

• New tasks will start this week
  • Nexus-5-L manual smoke test
  • Nexus-5-L regression test
  • Flame-kk full test
  • Flame-KK Bluetooth PTS certification

• Pending tasks
  • Nexus-5-L automation test - Keyboard issue
  • Nexus-5-L MTBF - Keyboard issue, test not running smoothly.
  • Performance - will be on soon, link (VPN required)

V2.2 Milestone

• 2015/02/23 V2.2 FL
• 2015/04/06 V2.2 FC

• QA Shared calendar
  • You can add it to your Google Calendar view by clicking on this link and then clicking on the small blue "+ Google Calendar" button at the bottom, right hand of the screen.
  • If you are signed in with your @mozilla.com account, you can also access it this way:
    • In Google Calendar, you should go to Other calendars > Browse Interesting Calendars > More > Resources for mozilla.com > Public (9) > Public - FxOS QA Schedule > Subscribe

V2.2 Testing Scope

• Focus on bugs in agreed list.
• B2G V2.2 Overview Page for more info.
• Current feature-b2g=2.2+ 13 bugs
• Current blocking-b2g=2.2+ 108 bugs

V2.2 Testing by Type

Smoke Testing

Flame-KK On Track

• 2 blockers.
  • Bug 1121045 - [b2g] we need root access on the b2g builds ( 3.0 & 2.2) for flame-kk
  • Bug 1118711 - Email app fails to fully launch due to worker JS runtime problem (reported as "InternalError: too much recursion") breaking startup process

• Existing Bugs Breaking the Smoketests: Flame-KK
  • [Disabled] test_setup_and_send_active_sync_email.py - Bug 1028192 - Problems with ActiveSync accounts
  • [Disabled] test_receive_active_sync_email.py - Bug 1028192 - Problems with ActiveSync accounts
  • [Disabled] test_cost_control_data_alert_mobile.py - Bug 1061698 - Need ability to work in the shadow root/shadow DOM
  • [Disabled] test_call_log_all_calls.py - Bug 1096385 - Investigate failure in test_call_log_all_calls.py, call log is displayed erroneously

Nexus-5-L On Track

• Will start from 3/3/2015

• Daily Reports can be found here.
  • Manual - are available on Google Drive & fxosqa-report-watchlist.
  • Automation - are available on Github.IO & fxosqa-report-watchlist.

Functional Testing

Yellow

• QA focus - 42 bugs found.

• RTL (Marigold)
  • Health: Yellow
  • Test case coverage: 18/26 ready for testing
  • Details: 11 blockers -
    • all from this week except one DOM bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1082060 filed on the 17th but possible workaround
    • All blockers found here: http://mzl.la/1wpXqXA
• Privacy (Gerry)
  • Health: Yellow (Features can work)
  • Test case coverage:
  • Details: 2 blockers
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1111992
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1093688
• Cut, Copy, and Paste (Gerry, Edward, Teri, Eric)
  • Health: Yellow
  • Test case coverage:
  • Details: 3 blockers
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1130982
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1114509
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1125045
• NFC (Alison) absent today
  • Health: Yellow
  • Test case coverage: 100%
  • Details:
    • User story [bug 1102019] is done. I will start test run to verify it.
    • User story [bug 979158] plans to postpone to 3.0 since part of features not finished yet.
• Search app (Hermes)
  • Health: Yellow
  • Test case coverage: 100%
  • Details: Features have been landed.
    • 5 WIP blockers:
      • https://bugzilla.mozilla.org/show_bug.cgi?id=1131966
      • https://bugzilla.mozilla.org/show_bug.cgi?id=1125131
      • https://bugzilla.mozilla.org/show_bug.cgi?id=1131408
      • https://bugzilla.mozilla.org/show_bug.cgi?id=1132262
      • https://bugzilla.mozilla.org/show_bug.cgi?id=1133202
• App-grouping (Hermes)
  • Health: Green
  • Test case coverage:100%
  • Details: Features have been landed. No 2.2+ blockers for now.
• Task Manager (Hermes)
  • Health: Green
  • Test case coverage: 100%
  • Details: Features have been landed.
    • 2 WIP blockers:
      • https://bugzilla.mozilla.org/show_bug.cgi?id=1120974
      • https://bugzilla.mozilla.org/show_bug.cgi?id=1107139
• L-porting (Hermes)
  • Health: Green
  • Details:
    • Camera recording is still not ready, and its patch is waiting for review. https://bugzilla.mozilla.org/show_bug.cgi?id=1120780
    • PVT builds for 2.2 & master are all ready.
    • Regarding smoketest, we will perform once and then ask Marigold for daily run.
• CAF bugs (Eric)
  • Health: Green
  • Test case coverage: 100%
  • Details: no blocker.
• TCL bugs (Alison, Eric)
  • Health: Green
  • Test case coverage: 100%
  • Details: 1 blocker https://bugzil.la/1131261 r?
• Bluetooth (Eric, Teri)
  • Health: Green
  • Test case coverage: 100%
  • Details: BT team is working on issues. Start testing next week.
  • Currently new daemon code is pref off due to OTA problem. https://bugzilla.mozilla.org/show_bug.cgi?id=1134821 need to wait for this to be resolved
• Metrics (Shing)
  • Health: Yellow
  • Test case coverage: 60%
  • Details:
    • 3 Testing https://bugzil.la/1119009 https://bugzil.la/1021173 https://bugil.la/1037495
    • 2 WIP https://bugzil.la/1109422 https://bugzil.la/1126524
• 256M RAM memory optimization for WFVGA, 5MP. (Teri)
  • Health: Green
  • Test case coverage: 100%
  • Details: MTBF in progress. No blocker.
• Multicore performance (Edward)
  • Health:
  • Test case coverage:
  • Better performance on Multicore (App startup time for core apps is comparable to Android
  • Upgrade raptor environment to 0.6.0
  • Set up auto testing environment in TPE Lab
  • So far, we provide performance results include "Gallery, Message, Music, Contacts, email, FM, Camera, Video,Clock" (Flame + master)
  • Raptor still has some bugs such as touch event lose, we guess that binary format of Lollipop is changed. I need to identify this problem.
  • Follow up performance drop issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1132515)

Exploratory Testing

Flame-KK On Track

• 203 bugs found - includes every bug in bugzilla affecting v2.2, not limited to the bugs after branch day.
  • UNCONFIRMED, NEW, ASSIGNED, REOPENED - 96 bugs found.
  • UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 7 bugs found.
  • RESOLVED, VERIFIED, CLOSED - 107 bugs found.
  • RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 7 bugs found.

• Regression
  • UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 39 bugs found.
  • RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 189 bugs found.

Performance Testing

Flame-KK Yellow

• will be on soon, link (VPN required)
  • Target platforms - flame-KK, flame-L

Stability Testing

Flame-KK Red

• Flame-kk MTBF - less than 10 hours.

Nexus-5-L Red

• Nexus-5-L MTBF - Keyboard issue, test not running smoothly.

V2.2 Test Case Creation

On Track

• Moztrap link of V2.2 New Features suite.

Test Case Creation WIP

• [2.2-feature-qa+] & in-moztrap? (0 bugs) --> WORK DONE
• feature-b2g=2.2+ & in-moztrap? (0 bugs) --> WORK DONE

Test Case Creation DONE

• [2.2-feature-qa+] & in-moztrap+ (19 bugs) --> WORK DONE
• feature-b2g=2.2+ & in-moztrap+ (78 bugs) --> WORK DONE

V2.2 Testing Verification Status

Overall Verification Summary

Need Verification

30 Total; 0 Open (0%); 30 Resolved (100%); 0 Verified (0%);

Verification Done

1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);