B2G/QA/Smoke Tests

From MozillaWiki
< B2G‎ | QA
Jump to: navigation, search

High Severity Bugs

  • Any action that bricks the device.
  • Any action that causes the device not to come up again after reboot.
  • Any action that causes the device to crash and reboot.
  • The following Sanity Test Cases fail without a workaround:
    • [RIL] Correct Radio Interface Layer is installed
    • [OTA] Over The Air update is successful
    • [Homescreen] The Homescreen is viewable
    • [Dialer] Can make an Outbound Call twice
    • [Dialer] Can make an Inbound Call twice
    • [SMS] Can send an Outbound Message twice
    • [SMS] Can receive an Inbound Message twice
    • [Camera] Can take a photo
    • [Camera] Can view the photo taken
    • [Browser] Can browse for 1-2 minutes over WiFi

Note: The telephony tests are done twice to make sure that the process cleans up after itself when the first instance is over so that the second instance is also successful.

Google Spreadsheet that tracks daily test logs for Sanity Test Cases.

Escalation Strategy

  • When a High Severity Bug is found, it is reported in Bugzilla with the keyword "smoketest".
  • A notification email is sent by QAnalysts to the following mailing lists: mozilla-dev-b2g@lists.mozilla.org, dev-b2g@lists.mozilla.org, dev-gaia@lists.mozilla.org and b2g-release-drivers@mozilla.org
  • The IRC channel owners of #fxos are notified in their channel.
  • A bug is filed requesting OTA to be blocked.
  • A regression window for the root cause of the bug is identified.
  • The developer who authored the commit is needinfo?ed with a request to back out the commit as soon as possible.
  • The Whiteboard tag "[backout-asap]" is added to the bug.
  • The module owner is also needinfo?ed.
  • QA tracks the bug until the commit has been backed out. Bugzilla Query for the Whiteboard tag "[backout-asap]": http://mzl.la/1A8gtQZ
  • QA requests for a new nightly build to be created.
  • When a new build becomes available, QA verifies that the bug no longer reproduces.
  • If the bug no longer reproduces, the bug requesting OTA to be reopened and a request is made to enable OTA again.

Smoke Test Blockers

Bugzilla Query for the keyword "smoketest": http://mzl.la/17ZCuek

Escalation Strategy

  • When a Smoke Test Blocker is found, it is reported in Bugzilla with the keyword "smoketest".
  • The QA Team is notified in the the #fxosqa IRC channel.
  • A regression window for the root cause of the bug is identified.
  • The developer who authored the commit is needinfo?ed with a request to back out the commit as soon as possible.
  • The Whiteboard tag "[backout-asap]" is added to the bug.
  • The module owner is also needinfo?ed.
  • A Smoke Test report email is sent daily by the Firefox OS QA team to the following mailing lists: fxosqa-report-watchlist@mozilla.org and b2g-release-drivers@mozilla.org
  • QA tracks the bug until the commit has been backed out.
  • QA requests for a new nightly build to be created.
  • When a new build becomes available, QA verifies that the bug no longer reproduces.

Test Automation

How to run test automation locally

1. Clone the gaia repository or pull the latest changes 

 git clone https://github.com/mozilla-b2g/gaia.git

2. Change the branch to the correct version that will be tested example: 

 git checkout v2.0

3. Flash your phone with an engineering build. (First make sure that ADB is enabled on your device.)

4. Point at the correct ports. 

 adb forward tcp:2828 tcp:2828

5. (Needs to be done once per setup.) Create a python virtualenv: 

 virtualenv .env     //--system-site-packages -- shouldn’t need

6. Enter by 

 source .env/bin/activate

7. Go into the tests directory. 

  cd [gaia repository]./tests/python/gaia-ui-tests

8. Install gaiatest. 

 python setup.py develop

OR 

 ./setup.py develop

9. (Needs to be done once per setup.) Copy the testvars_template.json to testvars.json in */gaiatest. 

 cp gaiatest/testvars_template.json gaiatest/testvars.json

10. (Needs to be done once per setup) Open the testvars.json file within your repository; bottom of the file, just before the last bracket, add (with the first comma): 

 ,"acknowledged_risks": true, "skip_warning": true

11. Run the Smoketest set: 

 gaiatest --testvars=gaiatest/testvars.json --address=localhost:2828 --type=b2g+smoketest 
 --timeout=20000 --html-output=results/results.html --restart gaiatest/tests/manifest.ini

Smoke Tests by Functional Area

Google Spreadsheet that tracks daily test logs for Smoke Test Cases.

Communications

Dialer

  • 1298 Verify an outbound call can be made and that the Call Volume can be adjusted
  • 6076 Verify that touch tones work using Mozilla's conference line.
  • (automated) Make an outbound call
  • (automated) Receive an incoming call with locked screen.

Messages

  • 10742 Send an MMS with picture attached
  • 1322 1293 (automated) Launch the Messages app and send an SMS to the same phone number. Verify that the SMS is received.
  • 10743 (automated) Launch the Messages app and create an SMS with some text. Attach an image by opening the Camera app and taking a photo. Send the SMS to the same phone number. Verify that the SMS is received. Tap on the attachment. Save the opened image on the SD Card and verify that there are 2 images.
  • 10742 (automated) Launch the Messages app and create an SMS with some text. Attach an image by opening the Gallery app and cropping an existing image. When the image has resized, tap on it to ensure that it is displayed. Send the SMS to the same phone number. Verify that the SMS is received. Tap on the attachment. Verify that the attached image is displayed.

SIM

  • 14361 Dialer Send SIM control codes for pin change
  • 10739 Send MMS to secondary SIM card (Auto retrieve on with roaming)

Contacts

Emergency Call

  • 15186 (automated) A regular phone number should not be dialed out on the emergency dialer

Telephony/Networking

WiFi

  • 6075 Verify the user can connect to WPA/PSK & WPA2/AES WiFi access points
  • 1327 (automated) Load a website via WiFi connection
  • 6075 (automated) Verify the user can connect to WPA/PSK & WPA2/AES WiFi access points

Cellular Data

Cost Control

  • 8938 (automated) Check if the data alert is triggered after 1-2 minutes of browsing over cellular data

Bluetooth

  • 6071 Verify that the user can enable Bluetooth and pair to a Bluetooth enabled computer/another device.
  • 6070 Pair and receive an incoming call using bluetooth
  • 9346 Music can be played and stopped remotely by user
  • 6071 (automated) Verify that the user can enable Bluetooth via Settings, rename the device and is discoverable by other bluetooth-enabled devices nearby.

FM Radio

  • 2461 (automated) Verify that headphones are plugged in and the FM radio has tuned in to the default channel frequency (lower bound). Flick up the frequency dialer a few times and check that the FM radio has tuned in to a higher default frequency.

Media

Camera

  • 2460 Verify the user can take a picture with camera from lockscreen.
  • 14789 Take pictures using Face recognition and Tap to Focus
  • 2477 (automated) Launch the Camera app, switch to video mode, record 3 seconds of video, check that the video has been saved on the SD Card
  • 1325 (automated) Launch the Camera app, take a photo, check that the thumbnail is visible and that the photo has been saved on the SD Card. Repeat this action twice more.

Video

  • 2478 Play the video you recorded. Check for video and sound
  • 2478 (automated) Launch the Video app and wait for the thumbnails to display. Tap on the first video and when it starts playing, verify that the elapsed time is greater than zero. Verify that the correct video name is displayed in the app toolbar.

Music

  • 4031 Artist View - Verify the user can play a single song.
  • 4031 (automated) Launch the Music app and verify for the tiles to display. Tap the Artists tab, tap play and verify that the music starts playing. Tap stop and verify that the music stopped playing.

Gallery

  • 14645 Test you can view a thumbnail in large mode
  • 14645 (automated) Check that photos can be viewed in portrait and landscape mode if the phone orientation is changed
  • 1326 (automated) Tap first image to open full screen view, then flick forward and backward to navigate.

Systems Front End

Browser

Homescreen

  • 14644 Battery Charge indicator given when charging
  • 6117 Verify the user can delete a packaged app
  • 1333 Marketplace - Verify the user can launch market installed hosted apps.
  • 5856 Install and launch Facebook and Twitter apps from Homescreen Search
  • 6116 Install and launch a packaged app
  • 6786 Verify a market installed hosted app can be installed and removed.
  • (automated) Install an app and verify that its icon appears on the homescreen. Activate the edit mode, tap on the (x) to start delete process, and tap on the confirm delete button. Check that the app is no longer available.
  • 6117 (automated) Install a packaged app and verify that its icon appears on the homescreen. Activate the edit mode, tap on the (x) to start delete process, and tap on the confirm delete button. Check that the app is no longer available.
  • (automated) Install an app and verify that its icon appears on the homescreen. Launch the app and verify that it is visible.
  • 6116 aster/tests/python/gaia-ui-tests/gaiatest/tests/functional/homescreen/test_homescreen_launch_app_packaged.py (automated) Install a packaged app and verify that its icon appears on the homescreen. Launch the app and verify that it is visible.
  • 1317 (automated) Move an app. Verify that the move activates the Edit mode. Exit the Edit mode and verify that the app has moved into the correct position.

Rocketbar

  • (automated) On the homescreen, bring up the context menu and select Add Collection. Verify that the collection is added to the homescreen. Tap the collection and select an app to save to the homescreen. Verify that the app was added to the homescreen.

Task Manager

  • 2462 (automated) Launch the Clock and Calendar apps. Long press the Home button to get the cards view and swipe to the previous app. Tap the card view of the app to bring it up again.

First Time Use

  • 6119 (automated) Launch the First Time Use tour, select en-US as the language, enable data if cellular network is present, connect to WiFi, set timezone if no cellular network, disable Geolocation, import contacts from SIM Card, skip Firefox Accounts, enable telemetry data collection, enter a dummy email address and skip the device tour.
  • 6119 (automated) Go through FTU with default values and walk through the tour. Check that you can go back a step and then continue till the tour is finished.
  • 6119 (automated) Go through FTU and verify that the Status bar is displayed throughout.

Graphics

Systems Platform

Lockscreen

  • 15243 Verify Lockscreen Display after timeout
  • 6784 (automated) Lock the device and then unlock it to homescreen. Verify that the homescreen is displayed.
  • 2460 (automated) Turn off the Camera geolocation prompt. Lock the device and then unlock it to camera. Verify that the Camera app is displayed.
  • 2460 (automated) Turn off the Camera geolocation prompt. Set a passocde for the lockscreen and enable it. Lock the device and then unlock it to camera. Verify that the passcode is needed. Verify that the Camera app is displayed. Take a photo and verify that it is stored on the SD Card.
  • 1330 (automated) Verify that menu options are displayed when the power button is long pressed.

OTA

  • 2313 System Updates - Success over Wifi

Multimedia Platform

Storage

  • 6072 Verify USB Mass Storage Functionality is correct
  • 6072 (automated) Toggle USB Storage on and off in the Settings app and verify if USB Mass Storage functionality is correspondingly enabled or disabled.

Productivity

Email

  • 10744 Notification for E-mail will be display
  • (automated) Connect to WiFi. Launch the email app. Setup a basic Gmail account with name, email address and password. Verify that the email app's header and toolbar area are visible. Verify that the account has emails after it has synced.
  • 6113 6114 (automated) Connect to WiFi. Launch the email app. Setup an IMAP account. Verify that the email app's header and toolbar area are visible. Verify that the account has emails after it has synced. Send to an email to oneself, wait till the new email is received and verify that it has the expected subject.
  • 10744 (automated) Connect to WiFi. Launch the email app. Setup an IMAP account. Wait for the account to sync. Exit the email app. Send an email to this IMAP account from a separate email client. Wait for the email notification to be displayed. Expand the notification bar and tap the notification. Wait for the email app to launch and verify that the email has been received.

Clock

  • 1772 [Alarm] Test that when the alarm triggers, the phone vibrates and an audible alarm is triggered
  • 1772 1775 (automated) Create a new alarm, give it a label and save it. Check that the confirmation message appears with a countdown. Edit the alarm and set it to trigger 1 minute from the current time. Turn off the alarm and turn it back on. Press the Home button and wait for the alarm to trigger. Stop the alarm and check that the Clock app closes and the Home screen appears.

Calendar

  • 6118 (automated) Can create a new event and view it on all calendar views: month, week, day

Marketplace

  • 1332 Marketplace - Verify the user can install market installed hosted app.
  • (automated) Verify that the Marketplace icon exists on the homescreen. Launch the Marketplace app and verify that it is displayed.
  • (automated) Launch the Marketplace app, search for a privileged app, install it and verify its icon is present on the homescreen.

Firefox Accounts

  • 13006 Successful Sign up in Settings
Retrieved from "https://wiki.mozilla.org/index.php?title=B2G/QA/Smoke_Tests&oldid=1184710"