Peptest

Overview

Peptest is an automated testing framework designed to test whether or not the browser's UI thread remains responsive while performing a variety of actions. Tests are simple JavaScript files which can optionally import Mozmill's driver to manipulate the user interface in an automated fashion.

The code is currently located at github. If you would like to help you can send pull requests, or ping ahal or mcote on IRC for ways to get started. Peptest is also mirrored to Mozilla Central. Firefox tests can be added to the tests directory (don't forget to also add them to the Test manifest).

For a longer description of what "unresponsiveness" actually means, see ted's blog post.

The Peptest wiki page contains project-related information, including current status and plans for future work.

Using Peptest

Running tests

Standalone

Currently tests are run from the command line with python. Make sure you have pip and virtualenv installed.

Create and activate a virtualenv, then run the command
```
pip install peptest
```

Run the command (use --help for a full list of options)

peptest --binary <path_to_binary> --test-path <path_to_test_manifest>

Alternatively if you will be modifying peptest, you can install peptest from the source repositories.

Clone the mozbase repository.
Create a new virtualenv and run 'python setup_development.py' to install all the packages.
Clone the Peptest repo and install peptest (python setup.py develop)

Run the command (use --help for a full list of options)

peptest --binary <path_to_binary> --test-path <path_to_test_manifest>

From within mozilla-central

After building, go to your obj dir and type make peptest. This will run all tests in proxy mode, i.e., without accessing the network.

Test manifest

All parameters are optional except for the --test-path parameter. This parameter should contain the path to a ManifestDestiny manifest that lists all the tests that will be run. Manifests should be of the following format:

# test paths are relative to the manifest
[test1.js]
[test2.js]
[foo/bar/test3.js]
# this test is disabled and won't be run
[test4.js]
disabled = "Doesn't work, see bug 123456"

Alternatively, the --test-path parameter can point to a single test which might be useful for debugging purposes.

Fail threshold

One more thing that can be specified from the manifest is the failure threshold. When each test finishes, the Peptest harness will aggregate all of the unresponsive times into one metric that can be used for regression detection. If you specify a failThreshold for a test, it will only be reported as a failure if its metric is higher than the threshold.

# no failThreshold, this test will fail if there is any unresponsiveness whatsoever
[test1.js]
# these tests will only fail if their metrics are greater than their failThresholds
[test2.js]
failThreshold = 42
[test3.js]
failThreshold = 500

Test format

Tests are simply JavaScript files that will be executed in chrome space. This means they have access to any of the APIs that an extension would normally have access to. In addition to this, they also have access to Mozmill's driver for convenient automation of Firefox/Thunderbird's UI.

It is helpful to think of tests as a series of actions. That is, the UI thread will only be checked for responsiveness while an action is currently happening. This ensures that we are only testing the actions that we care about, and that the test isn't overrun with noise generated during setup or teardown. To accomplish this, tests call a function called performAction(name, func) which takes in two parameters: an action name and a function pointer whose contents consist of a single action.

The following example test will make sure that the browser remains responsive while opening a page and while searching in Google.

// import Mozmill and initialize a controller
Components.utils.import('resource://mozmill/driver/mozmill.js');
let controller = getBrowserController();

// Create our first action which will open Google
pep.performAction('open_google', function() {
  controller.open('https://google.com');
  controller.waitForPageLoad();
});

// stuff not inside a performAction() call won't be tested for responsiveness
let textbox = findElement.ID(controller.tabs.activeTab, 'lst-ib');
let button = findElement.Name(controller.tabs.activeTab, 'btnK');

// Create our second action which will perform a search in the google searchbox
pep.performAction('search_google', function() {
  textbox.sendKeys('foobar');
  button.click();
  controller.waitForPageLoad();
});

For documentation on using Mozmill's driver, see the Mozmill Reference Desk.

Note that using Mozmill may be convenient but is not required. Here is another example that tests resizing the window without Mozmill.

// getWindow is a convenience function exposed to tests
// It uses nsIWindowMediator to get the current window
let window = pep.getWindow();
let width = window.outerWidth;
let height = window.outerHeight;

pep.performAction('resize_by', function() {
  window.resizeBy(100, 100);
});

pep.performAction('resize_to', function() {
  window.resizeTo(800, 600);
});

// Tests should clean up after themselves
window.resizeTo(width, height);

See the existing Firefox peptests for more examples.

Log format

Peptest uses the mozlog module to standardize on the logging format.

PEP TEST-START | test1.js
PEP TEST-PASS  | test1.js | fail threshold: 0.0 < metric: 0.0
PEP TEST-END   | test1.js | finished in: 1536 ms
PEP TEST-START | test2.js
PEP WARNING    | test2.js | loadPage | unresponsive time: 116 ms
PEP WARNING    | test2.js | loadPage | unresponsive time: 61 ms
PEP WARNING    | test2.js | loadPage | unresponsive time: 275 ms
PEP TEST-UNEXPECTED-FAIL | test2.js | fail threshold: 90.0 < metric: 92.802
PEP TEST-END   | test2.js | finished in: 8022 ms
PEP TEST-START | test3.js
PEP WARNING    | test3.js | scroll_menu | unresponsive time: 121 ms
PEP TEST-PASS  | test3.js | fail threshold: 15.0 < metric: 14.641
PEP TEST-END   | test3.js | finished in: 2137 ms

When a responsiveness measurement runs over 50ms, a TEST-UNEXPECTED-FAIL message is output. The second column is the name of the test file, the third column is the name of the particular action that was being performed and the last column is the time in milliseconds that the event took to process.

When the test is finished, the total run time and a metric are output. The metric is explained in the next section.

Metric

The metric is a number that is calculated based on all accumulated unresponsive times. The function used to calculate the metric is

# values is a list of unresponsive values that occurred during the test
def get_metric(values):
    return sum([x*x / 1000.0 for x in values])

The metric is useful for determining whether a test has regressed or not. For example if a test used to have a metric of 56.43 then all of a sudden has a metric of 193.42 we know a regression has happened.

Important: You cannot compare the metric from one test to that of another. Tests can be of an arbitrary length and therefore it is impossible to conclude that a test with a metric of 10 is more responsive than a test with a metric of 1000. You must only compare metric values to previous metric values of the same test.

Proxying to local files to avoid network access

Peptest can be configured to proxy all network requests to local files. There are three related options:

--proxy: This option must be given with the location to a text file specifying what URLs should be proxied. The file should have one or more lines in the form <url> [<options>]. See testing/peptest/tests/firefox/server-locations.txt for an example (as used in buildbot).
--server-path: This must be set to the path to a directory serving as the root of the local web server.
--proxy-host-dirs: This indicates that the files in the web server root should be organized according to URL. Without this option, the same content, as found in the server path, is served for all proxied URLs. With this option, the web server root should have one directory per URL. See the server files in the repo for an example of how the server root should be structured when --proxy-host-dirs is given (as used in buildbot).

Peptest Statistics

Peptest is run automatically as part of buildbot on all pushes to try, mozilla-inbound, and mozilla-centric. The results are gathered and displayed for analysis. (This is a temporary URL; eventually they will be folded into the next version of the graph server/perfomatic)

API reference

There is a Peptest API that gets injected into the scope of every test under the "pep" namespace. Each test can use the following methods:

General methods

pep.performAction(name, function)

Performs an action during which responsiveness measurements are recorded.

name: The name of the action to run.
function: A function to handle running the action.

pep.getWindow(type)

Returns a reference to the current window.

type: A string specifying the XUL windowtype of the window type to return; the default is "navigator:browser".

pep.sleep(milliseconds)

Unfortunately, UI automation is very dependent on timing and sometimes it is necessary to sleep; however, you should avoid this method if you can.

milliseconds: The number of milliseconds for which to sleep.

Logging methods

These log messages to the console.

pep.log.debug(msg)

msg: The message to output.

pep.log.info(msg)

msg: The message to output.

pep.log.warning(msg)

msg: The message to output.

pep.log.error(msg)

msg: The message to output.