Web Apps integration/TestPlan

From MozillaWiki
Jump to: navigation, search

Overview

Feature Release Target Dev Lead QA Lead Dev Status QA Status Health
Web Apps Integration Firefox 15 Felipe Gomes Jason Smith Landed In Progress OK

Feature Summary

This feature focuses on being able to find web applications to install on Firefox through the home tab and installing and uninstalling these applications natively different operating systems. These installable web applications are essentially websites built in web technologies (e.g. HTML, CSS, JavaScript) that users can interact with through a chromeless shell in a native environment. These web applications intend to act exactly like native applications on the operating system. For a website to become a web application, a developer creates an app manifest for their website and hosts it on an origin (e.g. www.yourhost.com) where the website is located. Then, the app developer adds this manifest to a store (e.g. Firefox store) to allow the app to be installed to users' machines.

Users can then go to this store to install this application to their native machine. Note that installable applications can be paid or free. For paid applications, users will have to pay for the application through Paypal. Upon making the application payment, a user's machine receives a receipt to prove that the application is paid for. Then a user can install this paid application to their machine on Win XP+ or Mac OS X. See the native installation flow to understand how native applications are installed a user's machine on Windows XP+ and Mac OS X. After a user selects to install an application, a confirmation should appear asking the user to confirm installation of the application to a particular file location on the user's machine. Then, upon confirmation, the application should be installed to the user's machine. At this point, the user should understand where the application was installed on their machine.

After applications are installed, users can run these applications in their native environment in a chromeless shell both online and offline. If the application is paid, validation will need to take place using the receipt for the application to ensure that the user paid for the application. On windows, applications are typically ran as shortcuts from the desktop, but may also be ran from the start menu in executables. On Mac OS X, applications are ran through the /Applications directory typically. When an application is launched, a chromeless window starts using Firefox's webapp mode under the hood with the web application running in the shell. In the task manager in Windows, you would see that the application is process running on the system under the name of the web application. Within the shell, users can take actions within the application, such as logging in, clicking links within the origin of the application, playing a game, and more. Within the shell itself, users have a menu to allow basic application actions such as quitting, cut/copy, and more. When a user is done using the application, they can quit using the application.

If the user no longer wants the application installed on their machine, they can uninstall the application from their machine. For windows, uninstalling can take place either directly in the directory location of the application or through add and remove programs. On Mac OS X, uninstalling occurs by moving the application to trash and deleting the application from the trash bin. Upon uninstalling an application, all locally stored app data that previously created during installation should be cleared.

Testing Scope

Major Features

The scope of testing of this feature focuses on the following major features:

  • Home tab web apps integration
  • Installing native applications
  • Launching native applications
  • Using native applications
  • Uninstalling native applications

Software Qualities

  • Security
  • Resilience
  • Localization
  • Performance

Edge Cases

  • Browser crashes
  • Loss of internet connection (e.g. offline mode)
  • App state changes (i.e. free to paid, paid to free)
  • Invalid or no receipts for paid apps
  • Local vs. roaming windows profiles
  • System restore

Testing Strategy

The following testing approaches shall be used to assess the scope of testing stated above:

Development of Manual Test Cases: Manual test cases shall be created to test the major portions of features in the testing scope. Each test case shall specify a quality requirement (smoke or feature signoff). Applicable sets of these test cases (smoke and feature signoff) should be ran each beta release that are value-added to run to capture bugs.

Exploratory Testing of Marketplace Top Apps: Exploratory testing of using top native applications shall be conducted to ensure that our highest priority applications are supported on our platform. When exploratory testing these applications, a checklist below will help guide the direction of exploratory testing:

  • Does the application avoid using engine specific content?
  • Does the application show out of origin content?
  • Does clicking out of origin links that are not authentication go to the browser?
  • Does application pop-ups and platform integration work as expected?
  • Does flash and experimental features work as expected?

For problems discovered that involves a top app, a whiteboard marker of '[topapps]' shall be specified. Any problems with the app itself shall have evangelism bugs logged if the app is a top app.

Development of Value-Added Automation: Automation shall be developed for smoke test cases that carry the highest value add in comparison to the overhead to the build the automation. See the automation section for more details.

Sign Off Criteria

Aurora

  • All Desktop WebRT blockers are fixed
  • Smoke tests should pass on Win XP, Vista, 7 (32/64 bit), OS X 10.5 - 10.7 with no major issues
  • No significant regressions in Desktop Firefox (e.g. crashes due to this implementation)

Beta

  • Flagged polish issues are fixed
  • Smoke and important feature signoff tests should pass on Win XP, Vista, 7 (32/64 bit), OS X 10.5 - 10.7 with no major issues
  • No significant regressions in Firefox due to this implementation

Release

  • Same as Beta

Infrastructure Requirements

mozqa.com's firefox test cases section shall need to be turned into a web app to allow for testing the underlying platform when using apps with desktop. If new test cases for platform are necessary, then they will need to be uploaded as test cases to the mozqa.com test server.

For testing the manifests with the apps lifecycle, testmanifest.com shall be used to do this. Manifests shall be attached to test cases. Then, the tester shall have to copy and paste the manifest into an applicable testmanifest.com URL, save it, and install the application. From there, other test cases can be done as needed for the apps lifecycle (e.g. uninstall application).

Test Case Management

Test cases for this feature shall be tracked in MozTrap. In MozTrap, they will be flagged as a Desktop Firefox 15 feature in the product version tag. Additionally, smoke tests will be flagged with the "smoke" tag. Feature signoff test cases will be flagged with the "feature signoff" tag. Each of these test cases are broken up into categories such as functional, security, resilience, localization, and performance and are tagged as such.

Note: Ideas for test cases are documented here

Automation

Automation for this feature shall intend to target smoke tests that touch the apps lifecycle (e.g. install, uninstall) and the use of the application (e.g. can I open a pop-up?). Since add-ons are not intended to be supported in the web runtime, this automation then shall be built using Marionette. This framework is necessary to use as Marionette does not rely on add-ons to run, where as other frameworks such as Mozmill do rely on add-ons to operate correctly. In addition to Marionette, we will need to build a python library that allows common operating system interaction in OS X and Windows to analyze major parts of the apps lifecycle (e.g. was a webapp.json file created, was the webapprt exe file created). For launching this automation, the following process has been suggested:

  1. mozrunner (mozprocess) launch the process
  2. marionette communicates with the process, tests
  3. (there is also the python-side of the tests)
  4. mozrunner/mozprocess terminates process
    1. (as a backup, the CI system kills processes and reboots the system - might require improving mozrunner for this need)

First steps to getting this setup will require ensuring that marionette can be built with firefox, as this has not been tested yet. Upon completing the desktop integration with marionette, we will need to write a python library for common OS interactions. Upon completion on the python library, we will then need to build a proof of concept automated test utilizing marionette with the python library. This test should target the basic apps lifecycle steps (e.g. install app, launch app, use app, uninstall app). Following completion of the proof of concept, further automated tests can be continued to be built as needed to get automated coverage of smoke tests with desktop web apps.

Bugs

A summary of the current bugs in the system are found here. The below lists summarizes the bugs that need to be verified:

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Important References

Open Questions

Reinstallation of Paid Apps with a Different Account Question

(jsmith) What's the difference with a different account when reinstalling the same paid app?

Install App on Two Different Firefox Profiles

(jsmith) What is the expected behavior if I install an app on two different firefox profiles? Will one app overwrite the other entirely? Partly?

Retrieved from "https://wiki.mozilla.org/index.php?title=Web_Apps_integration/TestPlan&oldid=433152"