Sheriffing/How To/Recommended Try Practices

The TryServer is a critical piece of our infrastructure for ensuring that patches are suitable for landing prior to being pushed. However, proper usage is important to avoid pushing under- or over-tested patches. This page is intended to provide some recommended practices for finding the right balance.

Note that the guidelines below are not a substitute for using your best judgment. The guidelines below are intended to be general recommendations but not absolute musts.

1. In many cases, running all builds and tests is overkill and should be avoided. A full Try run consumes over 500hrs of machine time to generate a full set of builds and run the full suite of tests across them. This can have a real impact on wait times across resource-constrained platforms, including production runs.
2. If in doubt, run both debug and opt builds.
3. If the patch only affects one platform, only that platform should have builds and tests.
4. Be aware of which test suites are likely to be affected by your patch and avoid running others that won't.
   • For example, a devtools-only change is unlikely to break any tests outside of mochitest-dt and xpcshell.
5. If the code is cross-platform, a T-style run (build on all platforms, test on one) will find most problems while making best use of resources. (Use copy several filter changes with ./mach try chooser and copy and paste if necessary).
   • Linux64 is the best choice for the test platform as it is the least resource-constrained.
6. If you no longer need builds/tests from a run, cancel the remaining jobs.
   • Use the stop button on Treeherder's job details panel.
   • Unneeded jobs are a frequent source of frustration-inducing backlog.
7. Any existing jobs on a push can be retriggered without a new push.
   • Use the retrigger button on Treeherder's job details panel.
   • You can trigger as many additional runs as needed (i.e. trying to track down an intermittent failure).
   • Retriggering a build will cause all tests to be run again after the build completes.

High scores!