Performance/Pageloader

The pageloader component is the core of many of the performance tests that we run via Talos and otherwise, including Tp3, Tsvg, Tgfx, and others.

The core of the test captures the amount of time between when a page load is requested (that is, when loadURI is called on the content element) and when the 'load' event is fired on the content. Additionally, the harness can also time page render times separately.

Installation

If you have a build with --enable-tests, the pageloader component is built and installed by default; no further building is needed. Otherwise, read on.

The net result of the build are three files in the dist directory:

• dist/bin/components/tp-cmdline.js
• dist/bin/chrome/pageloader.jar
• dist/bin/chrome/pageloader.manifest

For an easy install in an existing firefox build (even a binary build such as a nightly), grab the pageloader and reftest files from here, and unzip in the dist/bin directory. The files will get placed into the components or chrome dirs as appropriate.

If you wish to examine or modify the pageloader, the code lives in mozilla/layout/tools/pageloader. You can either manually run make inside that directory in your objdir.

Running

The pageloader is driven by a manifest file and a set of command line flags. Running ./firefox -help shows the pageloader options, all of which start with "-tp". If these aren't present, then the pageloader components are not available or correctly installed.

-tp file

Required. file contains the manifest of pages to test; see below for more information.

-tpcycles n

Default: 5. Loop through the selected pages from the manifest n times.

-tpfilter regexp

Optional. Only include pages from the manifest that match regexp.

-tpstart n

Optional. Start on the nth entry in the manifest.

-tpend n

Optional. End on the n<i>th entry in the manifest.

-tpformat <i>f1,f2,...</i>

Default: js. Display results in the given formats. Supported formats are <i>text,tinderbox,js. See below for samples of each format.

-tpchrome

Run tests within the full browser chrome. Without this flag, pages are loaded in a plain toplevel window, without any additional browser functionality enabled.

-tprender

Run the render benchmark for each page in the manifest, along with timing page load. This will time how long it takes to redraw each page once it's already been loaded, and is most useful for measuring rendering speed.

-tpwidth width, -tpheight height

Run the tests in a window of width x height. Default: 1024x768.

-tpoffline

Force the browser into offline mode while running tests, to ensure that no network connections are created.

-tpnoisy

Dump the last loaded page to the console as the test progresses.

-tptimeout n

Default: none. Maximum amount of time to wait for a page to load, in milliseconds; if the page takes longer than this to load, abort.

-tpnoforcecc

Don't force cycle collection between each pageload. If this isn't done, pageload times can be skewed due to cycle collection kicking in while a page is loading. In normal browsing usage, CC is done during times when the user is idle.

The Manifest File

The manifest file is a simple text file with instructions on which URLs to load. Whitespace and other special characters in URLs must be %-escaped. Comments begin with #, and blank lines are ignored.

URLs are listed one per line, in the order that they should be loaded.

Tests may do their own timing, instead of relying on the automatic timing by the test harness. If a URL line starts with a "%", the test does its own timing. Such a test should call "tpRecordTime" in their JS and pass the number of milliseconds the test took.

Additionally, an include directive may be called to include another manifest.

Example manifest:

# This is a comment

# Load news.google.com
http://news.google.com/

# Load sometest.html in the same location as the manifest
sometest.html

# Load gfxtest.html in the same location as the manifest,
# and indicate that it will do its own timing
% gfxtest.html

# Then include morepages.manifest
include morepages.manifest

Note that to obtain stable numbers, it is generally best to set up a local http server with cached versions of all content that is to be loaded. However, measuring live http sites over the internet can be useful, as is using local files using file:///.

Output Formats

Four different output formats are supported by the pageloader harness, and one or more can be given as arguments to the -tpformat flag. Output is printed using the javascript dump() command; as such, the profile in use must have the "browser.dom.window.dump.enabled" pref set to true to see any output.

text

Human-readable format. Example:

============================================================
    Page                                     mean  stdd   min   max raw
  0 yahoo.com/www.yahoo.com/index.html         89     2    85   315 315,90,85,88,91
  1 msn.com/www.msn.com/index.html            342     5   336   938 938,336,347,338,346
  2 google.com/www.google.com/index.html       55     5    49   113 113,56,52,49,61
  3 baidu.com/www.baidu.com/index.html         46     3    43   951 951,43,44,44,51
Cycle collection: 1070
============================================================

js

Mostly-JSON-like format; useful for automated tools. Example:

([
({page:"yahoo.com/www.yahoo.com/index.html", value:88.5, stddev:2.29128784747792}),
({page:"msn.com/www.msn.com/index.html", value:341.75, stddev:4.815340071064556}),
({page:"google.com/www.google.com/index.html", value:54.5, stddev:4.5}),
({page:"baidu.com/www.baidu.com/index.html", value:45.5, stddev:3.2015621187164243}),
])

tinderbox

Legacy tinderbox reporting format. Example:

__start_tp_report
_x_x_mozilla_page_load,63.5,NaN,NaN
_x_x_mozilla_page_load_details,avgmedian|63.5|average|62.83|minimum|NaN|maximum|NaN|stddev|NaN
|0;yahoo.com/www.yahoo.com/index.html;89;88.5;85;315;315;90;85;88;91
|1;msn.com/www.msn.com/index.html;346.5;341.75;336;938;938;336;347;338;346
|2;google.com/www.google.com/index.html;54;54.5;49;113;113;56;52;49;61
|3;baidu.com/www.baidu.com/index.html;47.5;45.5;43;951;951;43;44;44;51
__end_tp_report
__start_cc_report
_x_x_mozilla_cycle_collect,1070
__end_cc_report

Example Commandlines

./firefox -P test -no-remote -tp pages.manifest -tpcycles 5 -tpoffline

Run using the "test" profile, without trying to contact an already running firefox (-no-remote). Read URLs from pages.manifest, run 5 cycles, and work offline.

./firefox -P test -no-remote -tp pages.manifest -tpstart 5 -tpend 10 -tprender -tpformat text -tpnoisy

Run pages 5..10 from pages.manifest, run the render benchmark after each pageload, and print the results as text. Also print each URL as it's loaded.

Bugs and Patches

Any bugs or patches should be submitted to the Testing:General component.