QA/TDAI/Meetings/2007-01-05
From MozillaWiki
« previous mtg | index | next mtg »
TDAI - Test Development, Automation, Infrastructure Discussion
- 10am, January 5th, 2007
- Dial-in:
- Dial Toll-free Number: 1-866-879-4799 (password: 369) or International Number: 650-903-0800 x91
- Then dial my conference number 245#
Contents
Agenda
- Status/plans for buildbot infrastructure
- Review reporting infrastructure
- Status of Test Extension
- Update on the Tp test tool
- Discussion test suite/harness/infrastructure inventory
- Discuss options for detecting server side bouncer problems like we had for 1509/2001
Attendees
- tracy, coop, marcia, juan, bc, martijn, alice, robcee, tony, timr, carsten
Discussion
Status/plans for buildbot infrastructure
- Linux VM doesn't have windowing env
- Robcee had requested the buildbot to be configured as a "build" spec VM. He didn't realize this meant no windows
- Options: use RPM?
- tinderbox "build" spec needs windowing env to run tinderbox tests [coop]
- Robcee and Coop with work on this with rhelmer <<== AI Robcee and Coop
- Next steps once VM is ready:
- Land buildbot config for Win and Linux from robcee's local config
- Get Makecheck tests running
- Setup Reftests - (results online on new tinderbox page)
- Setup Mochikit tests
- Currently these are rendering/display(?) regression tests
- Then JS tests
Review reporting infrastructure
- tinderbox sucks for double clicking on results
- eggplant results might be a good as a next step to capture and report results with the ability to drilldown
- Key things a database like litmus provides:
- determining for which tests were run, when they were run, last pass or fail, what OS they were rub on, what versions of the product were tested
- trending over time
- red/yellow/green dashboards
- Priority is on getting tests to report status to tinderbox page. Then collecting/presenting the data in more interesting ways
- Next steps:
- Tracy to get eggplant fully automated and tied into litmus <<== AI: tracy and coop
- Next steps:
Status of Test Extension
- Robcee sent David Joseph Hamp-Gonsalves list of requirements - not heard back
- Requirements from Rob's email:
- store user login (may not be necessary since litmus'll put a cookie on the user's machine anyway)
- run tests
- Select product,
- Platform,
- OS,
- ...
- testing group
- subgroup
- walk the user through each test in the group
- iterate through the steps to perform or even just display them
- present the expected results
- let the user enter pass/fail/unclear, notes and associated bug ids.
- submit the results
- David seems enthusiastic to help extend his current extension concept
- Tracy - have David meet us in Toronto? Good idea!
- Who should follow-up with David? <<== AI timr
Update on the Tp2 test tool
- graydon - asked Alice to run on trunk - broke
- We assumed, if it worked on 1.8.0 and 1.8.1, it should run on 1.9
- Bad assumption!
- Tp2 Currently runs on Window and we have results for 1.8.0 and 1.8.1
- OS X perf tests - [robcee]
- Need Mac/linux timing utils. Robcee knows of python tools for this
- Ideas from Schrep
- (left handwritten notes at work - will provide better details and priority later)
- Update current perf tests: Tp, Ts,
- competitive benchmarking
- platform specific
- more granular (not just high level user based)
Discussion test suite/harness/infrastructure inventory
- The purpose is to get better visibility into:
- Which test harnesses and infrastructure do the current tests use
- What is the status of development or consolidation of test harnesses
- What is the status of setting up major infrastructures
- Some important side purposes are
- Get a sense of scope and coverage for the test suites we have. This is a side purpose as understanding scope and coverage requires different and more detailed information.
- Get a sense of how usable and how well documented these harnesses are. This is a side purpose as this will be embedded in the status but really needs more details that should be placed elsewhere.
- Some tests, harnesses, and other resources we discussed
- necko harness
- nss suite
- nspr suite
- xpfe suite
- jssh harness - out of date - doesn't work on trunk
- Introduces event timing issues - possible incorrect ersults [rob]
- js/sec automation harness
- sec not automated
- timr and others seem to keep forgetting it is not automated %-}
- js framework is custom build by bc and is used for other things
- It uses the shell scripting language to perform a set of very modular steps:
- build/install browser on 3 platforms/ 3 branches
- create profile
- start browser
- execute web pages
- collect results
- It is a set of building blocks
- Plans
- Get this into src tree testing dir soon for reuse by others
- Need better reporting
- Need better ability to add tests
- It uses the shell scripting language to perform a set of very modular steps:
- sec not automated
- Other lists of test tools and references:
- from robcee:
- from marcia:
- lxr.mozilla.org/seamonkey/source/embedding/qa/testembed/ - Gecko API work from Dave Epstein that may be able to be re-purposed
Discuss options for detecting server side bouncer problems
- This is prompted by the major last minute problem we had for the 1509/2001 release
- Bug covering this: bug 364499
- One option is to automate server side verification
- We need a tool to get updates from AUS and verify the update is applied correctly
- Davel's update checker just gets the mar file from an ftp server and applies it. This does not test the AUS mechanism not the client side mechanism.
- A related issue is why people are getting multiple updates
- Next steps for main issue: bug 364499
- See how the 1509/2001 postmortem goes on Monday, Jan 8th. If this is still a high priority after that meeting we can see who else is willing to help create a test tool and then either lead or support this project.