B2G/QA/2.2/2015-03-09Status

V2.2 Highlight & Risk

• Highlight
  • Availability of Flame-L is unknown.
  • Current running tasks
    • Flame-kk manual smoke test
    • Flame-kk automation test
    • Nexus-5-L manual smoke test
    • Nexus-5-L regression test
    • Flame-kk full test
    • Flame-KK Bluetooth PTS certification

• Pending tasks
  • Nexus-5-L automation test - trial run, will verify the result in this week.
  • Nexus-5-L MTBF - trial run, will verify the result in this week.
  • Performance - will be on soon, link (VPN required)

V2.2 Milestone

• 2015/02/23 V2.2 FL
• 2015/04/06 V2.2 FC

• QA Shared calendar
  • You can add it to your Google Calendar view by clicking on this link and then clicking on the small blue "+ Google Calendar" button at the bottom, right hand of the screen.
  • If you are signed in with your @mozilla.com account, you can also access it this way:
    • In Google Calendar, you should go to Other calendars > Browse Interesting Calendars > More > Resources for mozilla.com > Public (9) > Public - FxOS QA Schedule > Subscribe

V2.2 Testing Scope

• Focus on bugs in agreed list.
• B2G V2.2 Overview Page for more info.
• Current feature-b2g=2.2+ 10 bugs
• Current blocking-b2g=2.2+ 95 bugs

V2.2 Testing by Type

Smoke Testing

Flame-KK On Track

• Existing Bugs Breaking the Automation smoketests: Flame-KK
  • [Disabled] test_setup_and_send_active_sync_email.py - Bug 1028192 - Problems with ActiveSync accounts
  • [Disabled] test_receive_active_sync_email.py - Bug 1028192 - Problems with ActiveSync accounts
  • [Disabled] test_cost_control_data_alert_mobile.py - Bug 1061698 - Need ability to work in the shadow root/shadow DOM
  • [Disabled] test_call_log_all_calls.py - Bug 1096385 - Investigate failure in test_call_log_all_calls.py, call log is displayed erroneously

Nexus-5-L On Track

• 1 blocker.
  • Bug 1118711 - Email app fails to fully launch due to worker JS runtime problem (reported as "InternalError: too much recursion") breaking startup process

• Daily Reports can be found here.
  • Manual - are available on Google Drive & fxosqa-report-watchlist.
  • Automation - are available on Github.IO & fxosqa-report-watchlist.

Functional Testing

Yellow

• QA focus - 42 bugs found.

• RTL (Marigold)
  • Health: Yellow
  • Test case coverage: https://wiki.mozilla.org/RTL

   5 P2 still ongoing, ETA 3/6

• • Details:

   TC: Keyboard, Setting, and Search are ongoing
   Bug: 14 bugs still unfix: http://mzl.la/1DTR6HJ
   

• Privacy (Gerry)
  • Health: Yellow
  • Test case coverage:
  • Details: 4 blockers
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1111992
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1136595
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1093688
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1137083
• Cut, Copy, and Paste (Gerry, Edward, Teri, Eric)
  • Health: Yellow
  • Test case coverage:
  • Details: 2 blockers
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1130982
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1136041
• NFC (Alison)
  • Health: Green
  • Test case coverage:100%
  • Details:
    • Full run testing now
    • One regression blocker: https://bugzilla.mozilla.org/show_bug.cgi?id=1120849 (patch wait for checkin)
• short_name support in manifest.webapp (Hermes)
  • Health: Yellow
  • Test case coverage: 90%
  • Details: Not fully landed. 3 WIP blockers
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1137498
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1137501
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1131338 (Resolved, but not uplift to 2.2)
    • Test cases in Moztrap will be done this week. (test data in qa-testcase-data repository)
• Search App (Hermes)
  • Health: Green
  • Test case coverage: 100%
  • Details: 3 WIP blockers
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1138631
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1134350
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1132262
• App-grouping (Hermes)
  • Health: Green
  • Test case coverage: 100%
  • Details: No 2.2+ blockers for now
• Task Manager (Hermes)
  • Health: Green
  • Test case coverage: 100%
  • Details: 2 Blockers
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1100610
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1107139
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1120974 (resolved not to 2.2)
• L-porting (Hermes)
  • Health: Green
  • Details:
    • Marigold has started to perform manual smoke test & regression test.
    • Camera recording is still not ready, and its patch is waiting for review. https://bugzilla.mozilla.org/show_bug.cgi?id=1120780
• CAF bugs (Eric)
  • Health: Green
  • Test case coverage: 100%
  • Details: Running regression test now.
• TCL bugs (Alison, Eric)
  • Health: Green
  • Test case coverage: 100%
  • Details: Running regression test now.
• Bluetooth (Eric, Teri)
  • Health: Yellow
  • Test case coverage: 100%
  • Details: Will start PTS certification test on flame-kk this week. Need to manually turn on bluetooth daemon(adb push an updated prop). Currently the new code with daemon is off by default due to bluetooth function break after FOTA, http://bugzil.la/1134821 a patch is under review.
• Metrics (Shing)
  • Health: Yellow
  • Test case coverage: 80%
  • Details:
    • 1 Blocker http://bugzil.la/1137063
    • 2 WIP http://bugzil.la/1130013, http://bugzil.la/1126524
• 256M RAM memory optimization for WFVGA, 5MP. (Teri)
  • Health: Green
  • Test case coverage: 100%
  • Details: Test in progress 50%, no blocker.
• Multicore performance (Edward)
  • Health:
  • Test case coverage:
  • Better performance on Multicore (App startup time for core apps is comparable to Android
  • Set up raptor for v2.2
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1132515
    • analyze and narrow down result from QA jenkins, http://bit.ly/1Glz30h. The problem should be between 4044 and 4045.

Exploratory Testing

Flame-KK On Track

• 223 bugs found - includes every bug in bugzilla affecting v2.2, not limited to the bugs after branch day.
  • UNCONFIRMED, NEW, ASSIGNED, REOPENED - 107 bugs found.
  • UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 14 bugs found.
  • RESOLVED, VERIFIED, CLOSED - 116 bugs found.
  • RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 8 bugs found.

• Regression
  • UNCONFIRMED, NEW, ASSIGNED, REOPENED & 2.2?, 2.2+ - 31 bugs found.
  • RESOLVED, VERIFIED, CLOSED & 2.2?, 2.2+ - 207 bugs found.

Performance Testing

Flame-KK Yellow

• will be on soon, link (VPN required)
  • Target platforms - flame-KK, flame-L

V2.2 Test Case Creation

On Track

• Moztrap link of V2.2 New Features suite.

Test Case Creation WIP

• [2.2-feature-qa+] & in-moztrap? (0 bugs) --> WORK DONE
• feature-b2g=2.2+ & in-moztrap? (0 bugs) --> WORK DONE

Test Case Creation DONE

• [2.2-feature-qa+] & in-moztrap+ (19 bugs) --> WORK DONE
• feature-b2g=2.2+ & in-moztrap+ (78 bugs) --> WORK DONE

V2.2 Testing Verification Status

Overall Verification Summary

Need Verification

30 Total; 0 Open (0%); 30 Resolved (100%); 0 Verified (0%);

Verification Done

1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);