QA/Fennec/Milestones/2.0/ReleaseTest/Beta3

From MozillaWiki
< QA‎ | Fennec‎ | Milestones‎ | 2.0
Jump to: navigation, search

Fennec 4.0 Beta 3 Test Results

The covered testing for this release will focus on smoketests, verifications and updates. For a list of new features coming in for Fennec 4.0, take a look at the Mobile Planning Page and for test coverage, look at our test plan.

Project contacts

PM: Stuart Parmenter (stuart)
Build: John O'duinn (oduinn)
QA: Aakash Desai (aakashd)

Schedule

Code Freeze
--
Builds hand to QA
--
QA test signoff
--

Build Location and IDs

  1. Go to the candidates folder in our ftp folders
  2. Go to the "4.0b3-candidates" folder
  3. Download the candidate:
    • For the n900, install the application catalog and update to the candidate build via your device App Manager.
    • For Android, download and install the actual build over any other instance

Test Coverage

Smoketests

Lite verification tests (or smoketests) will be first run when the builds are ready. Tests are available on litmus under the Fennec 2.0 Test Suite. In any case of failure in any Smoketest, blockers need to be raised ASAP before proceeding to the BFTs.

Please record your detailed results in the "Detailed Test Results" section below.

BFTs for 2.0 Features

BFTs are a more comprehensive set of litmus tests that cover functional testing of features that have landed on Fennec 2.0. Tests are available on litmus under the Fennec 2.0 Test Suite.

Please record your detailed results in the "Detailed Test Results" section below.

Here are the BFT's that need to be run:

  • Panning & Zooming (Screen Navigation)
  • Form Assistant
  • Sync

OS Integration

The Operating System Integration testsuite is a set of tests that target how Fennec works with the platform (i.e. Android, Maemo, etc.) and features of the device itself (i.e. Orientation, hard keyboard, soft keyboard, etc.). Tests are available on litmus under the Fennec 2.0 Test Suite under the "OS Integration Tests" testgroup.

Please record your detailed results in the "Detailed Test Results" section below.

Fixed Bugs Verifications

The following link contains a list of bugs that are fixed for beta 3 and need verifications:

L10n Checks

We don't have coverage of localized builds until the candidate builds are available, so we'll need to run through the tests within the Fennec 2.0 String Guide on the following locales:

  • German
  • Finnish
  • French

Updates

Test the update path for the released builds only, not the nightlies. (eg. 1.1->4.0b3 (maemo only), 4.0b1->4.0b3, and 4.0b2->4.0b3)

Fennec 2.0 Beta 3 can be downloaded from the Mozilla Beta Catalog.

Updates tests to perform for each build

Update Testing for bug 620584

12/21: There is a respin of Fennec Beta 3 build 2, that will pick up the changes to bug 620584. This changes the appname for fennec so the Android Market can host the app

  • Known Issues (Android Only)
    • There will be two instances of Mobile Firefox if the user has beta 1 or beta 2 previously installed. Both operate on completely different profiles
    • The app launcher may show two instances of Fennec when launching a url. They may be labeled both as Firefox, one being beta 1, the other beta 2.
    • AUS Updates for Fennec beta 1 and beta 2 will be disabled. Going forward, users will only get their updates from Android Market.
  • Testing (Android Only)
    1. Single install of Fennec beta 3 on android. Result: one instance of Firefox beta 3. Updates are disabled.
    2. Have an existing Fennec beta 2 on android. Install Fennec beta 3. Result: two instances of Firefox on android, with two different profiles. Both will work as expected. Updates are disabled.
    3. Have an existing Fennec beta 1 on android. Install Fennec beta 3. Result: two instances of Firefox on android, with two different profiles. Both will work as expected. Updates are disabled.
    4. Uninstall the existing instances in steps 2-3. Result: one instance of Firefox beta 3
    5. For steps 2-3, do not select a default browser, and launch a URL. Result: the app launcher appears with 2 instances of Firefox to choose from
    6. Run step 4, then step 5. Result: the app launcher appears with Firefox beta 3 to choose from

Mobile War Room Testing

We will be conducting a Testing and Feedback session for android users during the beta 3 release process. The purpose here is to spend some time using the product and looking at high level tests.

List of Testcases and Results

Test Results

Types of Tests Maemo Android
Smoketests
  • ahoza - DONE
  • Ioana - DONE
BFTs
  • ahoza - DONE
  • ashah - DONE
OS Integration
  • tchung martijn - PASS
  • Ioana - DONE
L10n
  • ahoza - DONE
  • Ioana - result
Updates
  • ahoza - DONE
  • tchung martijn - PASS
  • Ioana - result
  • anna - DONE
  • tchung - Pass

Detailed Test Results

Smoketests Results

HTC Desire [ioana]

  • Build: Mozilla/5.0 (Android; Linux armv7l; rv:2.0b8pre) Gecko/20101214 Firefox/4.0b8pre Fennec/4.0b3pre
  • Results: DONE
  • Notes:
    • the update appear both from alpha, and from 4.0b1/4.0b2 adding the property app.partner.test=test
    • bug 619308 Password can be seen after a failed login
    • ghost tabs appear when sharing a link through Facebook - not always

n900 [ahoza]

  • Build: Mozilla/5.0 (Maemo; Linux armv7l;rv:2.0b8pre) Gecko/20101214 Firefox/4.08pre Fennec/4.0b3
  • Results: PASS
  • Notes:
    • No new issues found

BFT Results

Nexus One [ashah]

  • Build: Build: Mozilla/5.0 (Android; Linux armv7l; rv:2.0b8pre) Gecko/20101214 Firefox/4.0b8pre Fennec/4.0b3
  • Results: DONE
  • Notes:
    • No blocking issues.
    • bug 619231 - Incorrect error message shown when nothing entered in sync login
    • bug 619584 - Google maps not properly displayed on fennec
    • bug 619587 - When searching a multiple selection combobox, the soft keyboard goes away after opening
    • bug 619712 - Cannot type in google.com

n900 [ahoza]

  • Build: Mozilla/5.0 (Maemo; Linux armv7l;rv:2.0b8pre) Gecko/20101214 Firefox/4.08pre Fennec/4.0b3
  • Results: PASS
  • Notes:
    • Form Assistant
    • Sync
    • Screen Navigation
      • No blocking issues found

OS Integration Results

HTC Desire [Ioana]

  • Build: Mozilla/5.0 (Android; Linux armv7l; rv:2.0b8pre) Gecko/20101214 Firefox/4.0b8pre Fennec/4.0b3pre
  • Results: DONE
  • Notes:
    • Known Bugs:
      • Blocking Beta3+bug 619308 Password can be seen after a failed login
      • bug 617555 System menu button closes the VKB
      • bug 615166 Search button in landscape mode has no caption
      • bug 608340 After dismissing the VKB once, you cannot bring up the VKB again by double clicking the awesome bar
      • bug 609600 Form helper restrict tab panel to one column
    • Unreproducible Issues:
      • Application freezes in Preferences after entering the sync section
      • Using Menu system button on HTC has different result than on other device: sometimes the url bar remains for this test case. Also pressing long menu system button will pop up the VKB and the URL remains always in front (test case fails )

n900 [martijn]

  • Build: Mozilla/5.0 (Maemo; Linux armv7l; rv:2.0b8pre) Gecko/20101214 Firefox/4.0b8pre

Fennec/4.0b3

  • Results: PASS
  • Notes:
    • Two new failures, filed the following bugs for it:
      • bug 619679 - [Maemo] Dismissing the virtual keyboard causes the top bar with "All pages", "Bookmarks" and "History" to disappear
      • bug 619347 - Tapping the URL bar once in portrait view does not bring up the virtual keyboard
    • Some other issues that I'm seeing (which I might have to file bugs on):
      • plugin.disable setting to false, all flash seems to be crashing (youtube videos crashing at least)
      • After a crahs, renewing the tab causes previous tab to load? Also seems to make the browser unusable for a while (need to investigate)
      • after switching tabs, the caret has become invible, when it had focus on a text field on a page
      • file picker causes painting issue on imageshack on the file input after opening.

L10n Results

Android [ioana]

  • Build: Mozilla/5.0 (Android; Linux armv7l; rv:2.0b8pre) Gecko/20101214 Firefox/4.0b8pre Fennec/4.0b3pre
  • Results: PASS
  • Notes:
    • No locale builds for Android. Used Multi Language build.
    • Deutch
      • untranslated words: downloads, add-ons, integer, boolean, string, desktop, true, false, true, name
      • large blue text pop ups due to language: block pop ups, offline storage, password manager
    • French
      • untranslated words: false, true.
      • large blue text pop ups due to language: block pop ups, geolocation
    • Finnish
      • large blue text pop ups due to language: block pop ups, password manager

n900 [ahoza]

  • Build: Mozilla/5.0 (Maemo; Linux armv7l;rv:2.0b8pre) Gecko/20101214 Firefox/4.08pre Fennec/4.0b3
  • Results: FAIL
  • Notes:
    • single locale builds won't lunch: bug 619375
    • multi language build contains strings properly localized for: French, Deutsch and Finnish

Updates Results

HTC Desire [ioana]

  • Build: Mozilla/5.0 (Android; Linux armv7l; rv:2.0b8pre) Gecko/20101214 Firefox/4.0b8pre Fennec/4.0b3pre
  • Results: PASS
  • Notes:
    • The Updates test are pass with one exception Test Case 13530 [Android Only][Update]Android Update with Auto Confirmation Setup
    • For all tests I've used the property app.partner.test = test in about:config


Droid 2 [Anna]

  • Build: Mozilla/5.0 (Android; Linux armv7l; rv:2.0b8pre)Gecko/20101214 Firefox/4.0b8pre Fennec/4.0b3
  • Results: PASS
  • Notes:
    • Test Case 13530 [Android Only][Update]Android Update with Auto Confirmation Setup is unclear/broken. I could not obtain the update, even setting the property app.partner.test = test in about:config.

Nexus One [Tchung]

  • 4.0b2 -> 4.0b3 en-US
    • Results: PASS
    • Notes: bug 619444 was filed against the beta-cck-test channel, and was fixed and verified. Need to a new String in about:config; app.partner.test = test in order to make it work.
  • 4.0b1 -> 4.0b3 en-US
    • Results: PASS
    • Notes: bug 619444 was filed against the beta-cck-test channel, and was fixed and verified. Need to a new String in about:config; app.partner.test = test in order to make it work.

n900 [ahoza]

  • Build: Mozilla/5.0 (Maemo; Linux armv7l;rv:2.0b8pre) Gecko/20101214 Firefox/4.08pre Fennec/4.0b3
  • Results: PASS
  • Notes:

n900 [martijn]

  • Build: Mozilla/5.0 (Maemo; Linux armv7l; rv:2.0b8pre) Gecko/20101214 Firefox/4.0b8pre

Fennec/4.0b3

  • Results: PASS
  • Notes:
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Fennec/Milestones/2.0/ReleaseTest/Beta3&oldid=274683"