Litmus:Web Services
Contents
Litmus Web Services Specification
Sending testcases to the client
Just use json: http://litmus.mozilla.org/json.cgi?testcase_id=101
Sending results to the server
Litmus will provide a web services interface to allow our automated testing infrastructure to submit test results to Litmus, where they can be tracked and reports of failing tests generated. For the time being, the testcases themselves will be managed as they currently are, outside of Litmus, but test metadata will be tracked in Litmus. In other words, each test will have a corresponding test id (we can do bulk imports of existing automated tests into Litmus), and the machines running the tests will send the results to Litmus.
To send results, testing machines would identify themselves to the server, describe the system configuration of the product under test (i.e. platform, buildid, locale, etc...), and submit a batch of results, giving the testcase id and the result (pass or fail) for each. In addition, testing machines can send logs to be attached to the results. Logs can be either general configuration information to be linked with the entire batch of results (environment variables, buildconfig), or detailed information about a specific test result only.
The following data would be sent for each communication with Litmus:
- Username
- Password
- Machine name
- Product
- Operating System (Note: this should correspond to one of the existing operating systems already in Litmus)
- Branch
- Build ID
- Build type (opt, debug, etc... freeform text)
- Locale
- (optional) Log(s) that relate to the entire test run
The following data would be sent for each result sent:
- Testcase ID
- Result status ('pass' or 'fail')
- Exit status (valid exit statuses are currently 'Exited Normally', 'Crash', and 'Timed Out')
- Duration
- Timestamp
- (optional) Comment
- (optional) Bug Number
- (optional) Log(s) that relate to this specific result
The following data would be sent for each log:
- Log type (coop has a long list types that we're using now, and more can be added easily)
- Log data - freeform text
Comments are appreciated on the proposed list of fields. --Zachlipton 11:27, 15 June 2006 (PDT)
Authentication
Authentication of the client poses a difficult problem. While it would seem that clients could send an encrypted version of their password to the server, Litmus (like Bugzilla) uses a random secret salt for additional security. Because of this, it is impossible for Litmus to know if a password is correct without having the plaintext version of the password the user entered. As such, we can't just send encrypted passwords over the wire.
User accounts that will be used for automation must be enabled by an administrator in the edit users interface. Enabling a user for automation assigns them a special authentication token that is used to identify themselves to the server. When submitting testcase data through the automation interface, the user sends their username and their token instead of the their normal Litmus password. This prevents automation accounts from being used for normal Litmus web login and allows authentication tokens to be shared amongst multiple users/machines.
Request Details
Testing clients will open an http connection to 'http://litmus.mozilla.org/process_test.cgi'. Clients must send via HTTP POST an XML document containing test results and metadata for submission. Results are formatted as described in the Test Result Format DTD. The parameter name will be 'data'.
All timestamps shall be in YYYYMMDDHHMMSS format.
Upon receipt, the server will process the test results. If a fatal error occurs, then no results will be added to the database. If some results are formatted properly, those results will be added even though other results may have errors.
Response
In the interest of keeping both the client and the server simple, the responses from the server are in plain text. After processing the results, the server will return (following the text/plain content type):
- The string 'ok' if all results were added to the database successfully.
- An error string beginning with the words 'Fatal error' if the result data cannot not be parsed or lacks required information.
- An error string beginning with the words 'Error processing result for test n' where n is the id of the test that resulted in the error. This line will be repeated for each result that contained an error.
Litmus APIs
Litmus provides JavaScript and Perl APIs to the web services interface, making it easy for automated testing harnesses to generate the proper XML file and submit it to the server.
JavaScript API
The JavaScript API is provided by the litmusReporter.js library, available for download from [1]. The API, essentially a direct port of the Perl API, provides functions to describe the system configuration, add test results, and serialize the data as XML in the proper format for submission to the web services interface.
Example code:
var litmus = new LitmusResults({username : 'foo',
password : 'bar',
/*optional*/ server : 'http://litmus.mozilla.org/process_test.cgi'
});
litmus.sysconfig({
product: 'Firefox',
platform: 'Mac',
opsys: 'Mac OS 10.4',
branch: 'Trunk',
buildid: navigator.buildID, // works on bonecho after 8/1
locale: 'en-US',
});
var res = new Result({
testid: 27,
resultstatus: 'pass',
exitstatus: 'Exited Normally',
duration: 2,
comment: 'test',
log: new Log({type: 'Foo', data: 'Bar'}),
})
litmus.addResult(res);
litmus.addLog(new Log({type: 'Abc', data: '123'})); // global log to be
// applied to all results
var xml = litmus.toXML();
Perl API
The Test::Litmus perl module automates the process of generating a properly formatted xml result file and sending it to the server. It provides an object-oriented interface to allow test scripts to easily submit results to Litmus.
The following example (perl) shows how to send your results to the server using the module:
use Test::Litmus;
$t = Test::Litmus->new(-machinename => 'mycomputer',
-username => 'user',
-authtoken => 'token');
$t->sysconfig(-product => 'Firefox',
-platform => 'Windows',
-opsys => 'Windows XP',
-branch => 'Trunk',
-buildid => '2006061314',
-buildtype => 'debug cvs',
-locale => 'en-US');
my $result = Test::Litmus::Result->new(
-isAutomatedResult => 1, # optional
-testid => 27,
-resultstatus => 'pass', # valid results are 'pass'
# or 'fail'
-exitstatus => 0,
-duration => 666,
-timestamp => 20051111150944, # optional (default: current time)
-comment => 'optional comment here', # optional
-bugnumber => 300010, # optional
-log => [Test::Litmus::Log->new( # optional
-type => 'STDOUT',
-data => 'foobar'),
Test::Litmus::Log->new(
-type => 'Extensions Installed',
-data => 'log information here')]
);
$t->addResult($result);
# $t->addResult($someOtherResult);
# etc...
# add log information that should be linked with
# all results (i.e. env variables, config info)
$t->addLog(Test::Litmus::Log->new(
-type => 'STDOUT',
-data => 'log data'));
my $res = $t->submit();
# $res is 0 for non-fatal errors (some results were submitted), and
# undef for fatal errors (no results were submitted successfully)
if ($t->errstr()) { die $t->errstr() }
Or, if you have already generated the xml result data manually, you may use the following to submit it to the server.
# $results contains the proper xml result data to send
use LWP::UserAgent;
my $ua = new LWP::UserAgent;
my $req = HTTP::Request->new(POST => 'http://litmus.mozilla.org/process_test.cgi');
$req->content_type('application/x-www-form-urlencoded');
$req->content('data='.$results);
my $res = $ua->request($req);
if (!$res->is_success) {
# an error occurred in the submission
}
my $content = $res->content;
if ($content =~ /^Fatal error/) {
# a fatal error occurred
} elsif ($content =~ /^Error processing result for test (\d+)/) {
# result $1 contained an error
}
