User:Tsk/Docs/QA Plan

Gross Initial QA plans/ Thoughts which needs to be correlated with all the other bits and pieces floating around MailNews:Home_Page and Thunderbird wiki pages. I need to start somewhere so here are the thoughts I've been having on how the QA should be done for Thunderbird. There are three main areas in which the QA effort should be divided.

Increase community testing

   - engage other communities - work in progress
       - because communities have mutual interests
               Linux distributions :
                   - OpenSuse (done)
                   - Ubuntu (resulst to be seen)
                   - Fedora (in progress)
                OpenSolaris - the people at Sun are interested and are willing to help:
                   - providing builds so the product can be tested
                   - using litmus/updating Litmus tests and doing litmus tests
               Local Communities
                   - started talking to geckozone (french gecko users) - got some interest but still need to figure out how to use them at best (ie litmus triagging)
                   - need to figure out what other communities I can easily engage here (Ask MozEurope about active communities)

   - have a better communication mean towards those wishing to test
       - explain what QA is so we could recruit a few users interested enough.
       - make sure are documents are up to date - and maintain them.
       - find the proper communications channels (experiences in progress)
           - at the moment for bug days and days I've used :
               - mozilla related news groups
               - Quality.mozilla.org

- How do we engage our users ?

           - for test days :
               - last time was one minute calls at the end of the release and it did not help
               - I'll send emails to people that provide builds so we can have testers easily.
               - Once I have the dev schedule for the release I'll make a Test Days schedule and will announce it :
                       - pages on the wiki
                       - events on quality.mozilla.org
                       - post to news groups
                       - email to interested community outside ours to see if they can relay the information
                       - engage geckozone - to see how that goes
                       - engage mozillazine
           - have more time to run the Litmus FFT for the next release and announcing way in advance that the test is going to be available
                       - communicate more on those tests

   - make sure the community test tools scale to the size of the community
           - litmus is sloooooooooow
           - how do we add things into litmus if we continue to use it ?
           - still not convinced by quality.mozilla.org UI is complex.
           - thunderbird-testers is a great idea - only has 158 members but it's been groing a bit. I think we should advertising it a bit more.
               (first idea would be to add a footnote on Gary's blog weekly post with a link to the email address)

bug database maintainance

   - faster triage
       - means more people see point 1 - means more capacity to read incoming bugs -> will be needed for RCs
       - means people with experience - experience does not come in a day so the earlier you start the better

(I plan on doing an interview with hansen to expose how he joined and how easy it was and etc ... to try to tell people that it's not that difficult to join - that it's fun and that you get something from it)

   - better quality of bugs
       - better documentation  (there are a few other things that can get logged, how is a mystery for me at the moment.)
       - buzilla helper for thunderbird that is  GSOC project which I would love to see coming to US (https://wiki.mozilla.org/Community:SummerOfCode09#Thunderbird) - what can we do to help there ?
       - How do we extract feedback left in Hendrix ?
               - shall we organize something like sherrfiff - have a list of Volunteers and make a schedule  like pereson A read week 7 and file bugs, person B week 8, or shall we continue the way it's done now ?
       - Shall we query crash stats on a weekly basis and create bug on that weekly basis ?
   - stats on bugzilla
       - Monthly report so we get a better idea how things are evolving.
       - same goes with crash stats should we force ourselves to query crash stats on a weekly basis and file bugs on a weekly basis

Automation

   - Unit Tests - how dow  we run that , on all builds ? A build a Day ? For all release builds ?
           - need to investigate or get input from development
   - Automated test breaking down TB in
       - UI
       - Local storage
       - Message treatment (red/unread/filter/composition etc ...)
       - Network protocols :
               - POP3
               - SMTP
               - IMAP
               - NNTP
               - HTTP
               - LDAP
               - SECU
           - break those down in  simple test using fake_server to create the test
           - make sure to prioritize which are should be worked on first (I would start with POP+SMTP)
       - How often do you run the tests - do the test results end up in Litmus ?
           - for all release but isn't that too late ?

   - UI with Mozmill
    - email read/reply
    - conf nntp
    - conf secu

   - Perf testing
       - against "real servers" ?
       - memory leak detection ? with extensions
       - old hardware ?
       - Launch ? dowload ? send ? filter

Extensions

I'de like to make sure that the most used extensiosn still work before we release

   - How do we build the list
   - What do we do when one is broken ?
   - Litmus ? MozMill ?