FirefoxSummit/2006/ProposedSessions/CommunityTestingAndLitmus/Notes
From MozillaWiki
Contents
Stats
- Now nearly 3500 testcases
- About 850 active testers out of the 2000 who have accounts.
- Not seeing much overlap between testdays, only a few people come back.
- Of the top testers, several community members keeping pace with MoCo staff testers.
- Dip in number of testers Aug/Sep -- one was Thunderbird which historically has less interest, and we had a lot of testdays around the same time, community getting burned out?
Test days
- Seneca testday 2006-11-03 with 35 testers. Harness that kind of support in an ongoing basis? Seneca students assigned to come, most probably won't come back.
- Trying to reward people with shirts and such, but we can't keep doing it to attract people
- School/university support
- Scheduling
- Make testing calendar more widely known, publicize dates better
- Don't schedule them at the list minute
- Send automatic email to new Litmus testers after they've done some testing to introduce Mozilla QA + provide resources to get more involved.
- More creative testdays--task oriented tests. The fun smoketest.
- Let people create their own testcases and groups specific to content and sites they interact with regularly. Mix and match existing testcases and new tests.
- Email out testday notifications to people who came to previous test days.
Manual test execution/Ways Litmus can help
- Partner testing
- Security flags for testcases
- One-off tests (emails, hendryx data, etc...)
- "Whiteboard" to track these tests, get them turned into real testcases
- Pull data from Bugzilla. List of bugs to be verified, etc... Make Litmus show other QA activities.
- Be better at vetting results + getting bugs found/filed in Bugzilla
- Display with checkboxes for admins to check off when results have been appropriately dealt with
- Programmatic regression analysis--determine when tests began failing and find the delta
- Daily digest email from Litmus, giving failures and stats from the day's testing -- Bug 360996
- Testcases in other languages
- Let the localizers take care of the results
- Do localizers really need tests in their own language?
- Create new l10n suite of tests that are sensitive to locale changes
- Easy way to tag tests/make a small list of tests to point people to around a specific area
Admin Interface Fixes
- Test group editing is foobar'd
- Adding testcases as a one-step operation again, not going through the hierarchy
Test Runs
- Right now a very freeform testing interface. Need a better way to target testers to what we need help with right now
- Adding an extra Test Run layer doesn't really add much functionality
- Homepage to point people towards what we want to test
- Limiting - test a particular build id on certain platforms
- How is a run different then a group?
- Run is a bucket of tests with a limited lifetime or permanent
- Make it simpler, point people to the build and the tests--pimp it to them up front
- Nomenclature - "run tests" and "test runs"
- Admin reporting
- Generate breakdown by subgroup, coverage data, etc...
- "What is Litmus" box on the side of pages, tips