B2G/QA/Gaia Test Plan/Email

From MozillaWiki
< B2G‎ | QA‎ | Gaia Test Plan
Jump to: navigation, search

General Information

Area Details
Team:
  • Developers:
  • Designer:
  • QA: Naoki Hirata

Test Plan

Introduction

This test plan is a test plan for testing the email app in the Gaia portion of b2g. This documentation will describe the testing strategy that will be used to validate the quality of the email app. Included are the various resources required to complete the testing.

Design Documentation/Background Research

Listed are the related documentation for testing:

Module Owners

PM 1.0.1 Requirements:

PM 1.1.0 Requirements:

Bugs tracking features:

Email Debugging info:

Specs:

QA Mapping

Module Owners

Automated Testing:

Repositories:

Tips and Tricks for testing:

OS Diagrams

Meetings

Schedule Milestones

Scheduling is based on the milestone listed on the Release Management wiki for B2G Landings:

Features that are required to land are listed here :

Lower priority bugs that may or may not make it into the milestones:

Hardware Requirements

  • A desktop machine capable of running ADB is necessary for testing for getting debug information and flash the device.
  • Mobile devices listed
  • Optional: Build machine to build your own gaia/gecko
  • Optional: Build OS Storage machine to make it easier to find regressions

Mobile Device

Test devices :

  • Leo
  • Inari

Backup test devices:

  • Otoro
  • Unagi

Software Requirements

System Testing

Mobile OS

  • B2G - Firefox OS 18

Accounts

  • See Test Accounts
  • Facebook Contact Import accounts also listed in the above link

Email Servers

  • Hotmail.com
  • Gmail.com
  • Mozilla.com

Strategy

  • Smoke Tests are run daily for sanity check on builds
  • Exploratory Manual
    • Modified Session Based testing around:
      • testing using outline; see Test Outline section
      • Bug fixes
      • Features implemented
  • Regression Testing based on Test Cases
    • Found via bugs
    • based on outline/feature/spec
    • Test cases will be tracked in MozTrap under "gaia" and "email"
  • Automation testing:
  • Performance Testing
    • Perfomatic
    • Note: we do need different specific email performance testing and there is a gap currently

Currently not tested:

  • Performance against different mobile data types

Test Deliverables

  • Test Cases for the User Stories before end of feature complete
  • Test Runs for each phase of the project
  • Smoke test runs on a daily basis
  • Automation Tests for future regression testing

Dependencies/Risk

  • Some Marionette features need to land before certain automation tests can be written. (ie swiping)
  • Test Runs w/ tests on the areas of change can only commence once the features have been complete.

Signoff/Exit Criteria

Feature Complete :

  • All User Story with leo+ blocking are closed and verified

Release:

  • All smoke tests and basic functional tests are ran without finding any blockers
  • All leo+ blockers that are fixed are verified

Test Outline

Note : This test out line is a brain storm of all possible conditions to consider for testing. The scope of the testing will be narrowed from this list based on design information, risks of breakage, etc.

Positive Smoke/Alpha Tests

UI

Functional

  • Conditional Functionality
    • Protocols:
      • IMAP:
        • Send
          • cc, bcc, to
            • interaction with contacts
            • verify date/time
          • sent folder copy
          • attachments
          • Cancel/Discard
          • Cancel/Save Draft
          • Home/Invoke multiple draft emails
        • Receive
          • cc, bcc, to
            • interaction with contacts
          • attachments
        • Move mail to another folder
        • Multi Move mail to another folder
        • delete mail
        • multiple delete mail
        • manual refresh
        • search
        • flag
        • multiple flag
        • read/unread
        • multiple read/unread
        • switching folders
        • Account Settings
          • Synchronize time
          • default account (currently busted)
          • Delete account
          • Add account
      • ActiveSync:
        • Send
          • cc, bcc, to
          • verify date/time
          • sent folder copy
          • attachments
          • Cancel/Discard
          • Cancel/Save Draft
          • Home/Invoke multiple draft emails
        • Receive
          • cc, bcc, to
          • attachments
        • Move mail to another folder
        • Multi Move mail to another folder
        • delete mail
        • multiple delete mail
        • manual refresh
        • search
        • flag
        • multiple flag
        • read/unread
        • multiple read/unread
        • switching folders
        • Account Settings
          • Synchronize time
          • default account (currently busted)
          • Delete account
          • Add account
  • Server
    • Hotmail Server
    • IMap Server
    • Gmail Server
    • Outlook Server

Beta Conditional

  • Virtual Conditions
  • Content Conditions
    • receive html version of email
    • check links in email
    • Phishing issues?
  • Physical Conditions
    • While Charged
    • While not Charged
    • Power Consumption
    • Performance
      • Speed of functions
      • Processor utilization
      • Memory utilization
        • Memory leaks?
        • Memory efficiency
    • Storage
      • Disk
      • Sim
  • Negative (Tests to Break) Tests
    • Security
      • Buffer Overflow (max characters)
    • removal of characters
    • Attachments
      • Uploading more attachment in size than server allows
      • Downloading more attachments than SDcard has space for
    • Account Setup:
      • IMAP:
        • invalid account
        • destination unreachable
        • wrong host name
        • wrong port
        • SMTP:
          • invalid account
          • wrong hostname
          • wrong port
      • Activesync:
        • invalid account
        • destination unreachable
        • wrong host name
        • https:// web site address (correct is m.hotmail.com)
  • Race Conditions
    • Lock conditions?
    • tooltips or other dialogs?
    • Network Conditions (Should have no impact)
      • WiFi
      • 3G
      • Bluetooth (Devices)
      • 4G?
      • SMS
    • Signal Conflict
      • Browser notifications
        • clicking in search field (form assistant + virtual keyboard)
        • Restart notification (ie. change in certain preferences in preferences panel )
        • Password notification
        • form manager notification
        • password manager
        • popup blocker
        • offline storage
        • geolocation manager respective to content window
      • Battery
        • Battery Low
        • No Battery
      • Calls
        • Incoming Phone Call
      • Other Apps
        • Calendar notification
        • Alarm Clock notification
        • Sleep notification
        • Screensaver / Lock (system)
        • EnergySaver (Fade) (system)
        • Other?
        • Music playing (should not affect)
        • plugins (youtube and other websites with different media types)
        • Extensions ( addons to fennec)
    • Buttons
      • Hard reset
      • Soft reset
      • Turn off phone
      • force quit
      • Other buttons being pressed on the phone (zoom, picture, power )
    • UI State Change
      • Zoom
        • while bringing up Keyboard
        • while dismissing the keyboard
      • Change Orientation
        • while bringing up keyboard
        • while dismissing the keyboard
  • Device/Platform Specific Issues
    • Portrait/Landscape Orientation
    • Camera (should not conflict?)
    • Microphone (Voice commands? Voice Recognition?)
    • Speaker (should not conflict)
    • headset (should not conflict)
    • Vibrate mode (should not conflict)
    • Volume (display conflict? need to test)
    • GPS
    • Print support?
    • Sync
    • Multiapp?
    • physical keyboard
      • visible
      • hidden

(Note : There may be some difference between maemo and android)

  • Localization
    • Double Byte Characters (Japanese, Chinese, Korean)
    • RTL Languages (Arabic/Hebrew)
    • High ASCII characters (Euro)
    • Date Input/Format (Euro/Japanese/Chinese)
    • Time Format (Chinese/Japanese)
    • Dialog check
    • Change of IME on Mobile OS
  • Help?
  • Performance

Usability

  • Usability
  • Accessibility (508 compliance)
Retrieved from "https://wiki.mozilla.org/index.php?title=B2G/QA/Gaia_Test_Plan/Email&oldid=648979"