Introduction

Grafx Bot is the new Halreftest! After meeting with members of the design team, Grafx Bot was selected as the new name for this project. Sean Martell is going to produce a robot icon/logo for it as well. bug 549416.

Grafx Bot is an extension being developed to enable crowdsource testing of hardware acceleration and layers on a greater variety of hardware combinations than Mozilla could reasonably expect to have access to internally. The tool will initially be promoted during QA testdays at periodic intervals, and may be promoted for wider distribution via some other channel after the graphics code has reached a desired level of stability. (what channel, and what criteria will we use to determine if it's ready for wider distribution?)

When a user invokes the Grafx Bot extension, it displays a welcome page which describes the purpose of the extension and provides a "Start" button. When the user clicks this, the extension will run a series of reftests which are bundled into the extension. When the tests are complete, the extension will post the test results, along with some system data, to a database at Mozilla, where it can later be analyzed.

Current Status [Mar 30, 2010]

• Extension code is now at http://bitbucket.org/jonallengriffin/grafxbot/ (mw22 update: this link is obsolete now. The code is now maintained at: http://hg.mozilla.org/automation/grafxbot )
• XPCOM component has been ported to Win, Mac & Linux
• Simple database views now available at http://brasstacks.mozilla.com/halreftest/data/
• Tool is being readied for internal testing within the Tools & QA teams
• Waiting on new icon/logo per bug 549416
• Waiting on privacy policy per bug 548884
• Need input from graphics team about which reftests we should be running
• There's an assertion which occurs when starting the test on Windows; filed as bug 556119

What It Tests

Grafx Bot is designed primarily to run reftests. According to developers on the graphics team, it should be possible using this method to achieve reasonable coverage of the relevant code. Currently Grafx Bot runs a subset of SVG tests, but this can be easily changed. Normally the current design runs reftests in the standard way, but will run them as follows on platforms that support hardware acceleration (currently Windows only):

1. hardware accelerated reftest test compared to accelerated reftest reference
2. hardware accelerated reftest test compared to non-accelerated reftest test

For #2 above, if a test fails, it may be due to small differences in the rendering engines used, which are not visible to the human eye. We really only care about rendering differences which are visible and obvious. To handle this case, failures of this type will be presented to the tester, with both images side-by-side, and the tester will be asked to determine if the images are visually identical or not. If they are, the test will be marked as passed.

All of the above tests are only designed to test rendering correctness. It is also desirable to test rendering performance, but there currently exists no easy way to do this. Graphics team developers have stated they may implement some type of system that could profile time spent in Cairo code, and if this is done, it could easily be added to Grafx Bot in order to provide performance metrics.

Collecting Test Data

All the data generated by Grafx Bot will be posted to a MySQL database at the end of the test. There are some simple views available at http://brasstacks.mozilla.com/halreftest/data/, and more can be added as needed.

The data collected includes all tests run and their results, and in case of test failure, the test images. We also collect various system stats to post along with the test results:

• OS
• Video card brand, name, device id, and vendor id
• Video RAM
• Video driver: files, date, and version
• Video display settings
• Firefox build date and version

Some system stats may vary depending on OS (e.g., it appears that the video driver on OSX is an intrinsic part of the OS, and thus doesn't have a distinct date and version).

We also allow testers to optionally provide comments at the end of the test, relating either to Grafx Bot itself or the hardware acceleration and layer features.

The User Interface

The current UI borrows graphic elements from other Mozilla properties.  Suggestions for improvements are certainly welcome!

The welcome page

The system information panel

The comparison screen

The 'Tests Complete' screen