Try Server for WebExtensions

WebExtension code is covered by unit tests, which are described at WebExtensions/Hacking#Unit_tests and WebExtensions/Contribution_Onramp#Testing. These tests can be run on the Try Server. Contributors with access to the try server can push try jobs to run tests.

For up-to-date documentation about the try server, see Pushing to Try in the Firefox Source Docs.

Try jobs can be scheduled via the web interface or via the "mach try" command.

The following platforms (-p) are of interest:

• win32,win64
• linux,linux64
• macosx64
• android-x86

Symbols for the following test suites are shown in the test results on Treeherder. Test suite identifiers for use with "-t" in "mach try" are listed below the main bullet points:

• M (mochitests) and M-1proc (mochitests in single-process mode)
  • mochitests (all mochitests, including many non-extension test suites). Not to be confused with "mochitest" (which only runs plain e10s mochitests).
  • mochitest-bc (browser chrome mochitests, i.e. test/browser/*.js; desktop only - Windows, Linux and macOS)
  • mochitest (plain mochitests, i.e. test/mochitest/*.html)
• X (xpcshell tests) and X-1proc (xpcshell tests in single-process mode)
  • xpcshell
• TV (test verification mode; same effect as "mach test path/to/your/test --verify" locally) (desktop platforms only; not available on Android)
  • test-verify
• ES (ESlint)

Pass --artifact to speed up builds (only if your changes do not require recompilation of C++/Rust code, such as Java (Android) and JavaScript-only changes).

Tests can be run in a debug build ("-b d"), an optimized ("-b o") or both ("-b do"). It is recommended to run both, because together they can catch more potential failures:

• Some tests are only enabled in one of the two build types because of intermittent/permanent test failures ("skip-if" in the .ini files that describe the tests).
• Debug builds usually run slower than optimized builds, and reveal weaknesses in tests (e.g. time-sensitive race conditions).
• Debug builds have additional checks (e.g. validation of extension API schema against the actual implementation).

Examples

By default, "mach try" is an alias for "mach try auto", which attempts to select some relevant tests. If unsure about what to run, "mach try" is recommended. To have more control over the tests to run, the newer "mach try fuzzy" or the classic "mach try syntax" can be used instead.

mach try fuzzy

Examples of "mach try fuzzy" commands (add "--no-push" to see the selected jobs without submitting to try):

• Run most relevant WebExtensions tests. This is recommended and was added in https://bugzilla.mozilla.org/show_bug.cgi?id=1856473

 ./mach try --preset webextensions

• Run relevant extension API tests on Android:

 ./mach try fuzzy -q "'geckoview '64-qr !gpu mobile/android/components/extensions toolkit/components/extensions

• (Note: if you get bash: !gpu: event not found, replace all exclamation marks (!) with "'!'")
• Run relevant extension API and AOM tests on Windows:

 ./mach try fuzzy -q "'windows11-64-2009-qr !wpt !gpu !msix browser/components/extensions toolkit/components/extensions toolkit/mozapps/extensions

If in doubt about what to use in the query (-q), run ./mach try fuzzy --no-push to launch an interactive program that displays the matched jobs.

mach try syntax

Examples of "mach try syntax" commands (add "--no-push" to see the generated try syntax without actually triggering the try job):

• Run all relevant tests. This is expensive and should only be used when you don't know which tests should be run, because it also includes many non-extension tests.

 mach try syntax -b do -p win64,linux64,macosx64,android-x86 -u mochitests,xpcshell,test-verify --artifact

• Run all relevant tests, without non-extension tests. The command with the `--tag=webextensions` flag uses your local checkout and .mozconfig configuration to identify the tests. Some Android-only tests may not run if your current configuration targets desktop.

 mach try syntax -b do -p win64,linux64,macosx64,android-x86 -u test-verify --artifact --tag=webextensions

• Run most extension tests (xpcshell, plain mochitests and browser-chrome extension tests (and many other mochitests/browser-chrome tests, but at least less expensive than just "mochitests"). The mochitest-bc test type is only used on desktop, so Android was omitted from the platform flag ("-p").

 mach try syntax -b do -p win64,linux64,macosx64 -u xpcshell,mochitest,mochitest-bc,test-verify --artifact

• Run extension tests on Android (requires a mozconfig file for Android; otherwise tests in mobile/android/components/extensions/test/ will not be run. The MOZCONFIG environment variable is used to select a different mozconfig file, so that you can develop for multiple platforms with a single checkout):

 MOZCONFIG=/path/to/mozconfig-for-android mach try syntax -b do -p android-x86 --artifact --tag=webextensions

It may take several seconds before a "mach try syntax" command with the "--tag=webextensions" flag completes. If you already know the directory that contains the test, it is recommended to use the directory instead of "--tag=webextensions", e.g.:

• Run tests in specific directories for desktop:

 mach try syntax -b do -p win64,linux64,macosx64 -u mochitests,mochitest-bc,xpcshell --artifact --and toolkit/components/extensions/test/ browser/components/extensions/test/

• Run tests in specific directories for Android:

 MOZCONFIG=/path/to/mozconfig-for-android mach try syntax -b do -p android-x86 -u mochitests,xpcshell --artifact --and toolkit/components/extensions/test/ mobile/android/components/extensions/test/

By default, a test that match any specified test tags ("--tag") or suite ("-u") will be scheduled. To only run tests that match all specified rules, include "--and" in the try syntax. When the "--and" flag is used, the set of tests is computed locally. To see what tests are going to be scheduled, add "--no-push".

Test failures

After running the tests, test results are displayed in Treeherder. Green tests mean success, orange means failure. Orange results can be resolved as follows.

• Your test or implementation caused a failure.
  • Fix your code and push a new try job.
• Your test or some other test failed unexpectedly (test timed out or cannot be reproduced locally).
  • Log in to Treeherder and click on the "Retrigger" button at your specific test to repeat the test run.
• Intermittent failures: An existing test without any relation to your code has failed.
  • Ignore the failure, especially if there is an existing bug report for it.

A set of tests can be split in different chunks to ensure that individual test tasks complete within a reasonable time frame. There is no obvious way to identify the chunk in which a test will be run, since it may depend on the build type, platform and the ordering of tests. To save server resources, you can run "mach try empty" to create an empty try job, and schedule specific jobs by selecting them in the Treeherder interface.