Please note, this is a STATIC archive of website developer.mozilla.org from November 2016, cach3.com does not collect or store any user information, there is no "phishing" involved.
Your Search Results

    web-platform-tests

    web-platform-tests is a regression test suite covering standard Web platform features. It is shared both among browser vendors and across browser platforms (such as Mozilla's Gecko and Servo), and is one of the primary means of verifying that browser implementations correctly implement the standards and are interoperable. Because of this, adding new tests to web-platform-tests (as opposed to Mozilla-specific Reftests, Mochitests, etc. suites) is encouraged whenever writing tests that might sensibly be run in multiple browsers.

    The web-platform-tests suite consists of two test types: JavaScript tests (to test DOM features, for example) written using the testharness.js library and reference tests (to test rendered output with what's expected to ensure that the rendering is done properly) written using the W3C reftest format.

    Mozilla has integrated web-platform-tests into mozilla-central, and has a two-way sync mechanism that allows developers to add/modify tests directly in the testing/web-platform directory in the Gecko source. (Changes are regularly synchronized between the Gecko source and the upstream GitHub repository.)

    Running tests

    Like other test suites, web-platform-tests are invoked using the mach command:

    ./mach web-platform-tests

    Individual tests can either be run by path in the source tree or by name (i.e. path under the server root). For example, the following two commands perform the same test:

    ./mach web-platform-tests testing/web-platform/tests/dom/historical.html

./mach web-platform-tests /dom/historical.html

    One can also pass the path to one or more directories to run all tests under that directory.

    Many of the same considerations that apply to mochitests/reftests apply to web-platform-tests; for instance, there may be tests that require focus.

    Running with e10s

    Presently e10s is not enabled by default on test runs. To enable it, pass the --e10s flag.

    Running under a debugger

    The tests can be run under a debugger using the same flags as for other test suites. For example: --debugger=gdb.

    The --pause-on-unexpected flag can also be useful to pause on the first failing test allowing you to attach a debugger and reload the test to debug it.

    Logging

    web-platform-tests use structured logging, and all the usual structured logging options are available.

    Writing tests

    General guides to the API for writing web-platform-tests are found on Test the Web Forward. However, there are some Mozilla-specific concerns detailed below.

    All web-platform-tests which are upstreamed live in testing/web-platform/tests. Since all files in this directory are synced with upstream, it is important that only tests that have a pass condition matching the specification and are suitable for running in multiple browsers are checked in here. Tests that cannot (yet) be upstreamed, such as those which use Gecko-specific features or are not in a state where they match the spec, can be checked in to testing/web-platform/mozilla/tests.

    Tests in testing/web-platform/mozilla/tests have /_mozilla/ prepended to their URL.

    Creating new tests

    Unlike many other test types, web-platform-tests do not have a human-written manifest. Instead, they take information about which files are tests directly from the test files themselves, found in a generated MANIFEST.json file that lives under testing/web-platform/meta. When creating a new test, it is important to update this file. The easiest way to do this is to use the helper command for creating new tests:

    ./mach web-platform-tests-create testing/web-platform/tests/dom/new_test.html

    This will create a test file with the testharness.js boilerplate, and start the manifest update process so that the new file appears in the JSON manifest. It will also launch the local Firefox build with the test file loaded. To create a reftest instead, pass --reftest to the mach command. Various other options for the test creation are available; see the command help for details.

    Alternatively, the manifest may be updated at any time by passing the --manifest-update flag to mach web-platform-tests. Note that by default this will also run the tests. That may be avoided by passing a path not containing tests to the mach command.

    Metadata files

    Because web-platform-test files are synced with upstream and do not use a human-written manifest, it's necessary to keep implementation-specific metadata outside the tests themselves. This is done in a metadata file under testing/web-platform/meta with the same path as the test, but an .ini extension e.g. testing/web-platform/tests/dom/historical.html has a metadata file under testing/web-platform/meta/dom/historical.html.ini. In addition, metadata can be set for a whole subtree of tests using a metadata file called __dir__.ini under the metadata root of that subtree (e.g. testing/web-platform/meta/dom/__dir__.ini).

    Despite the .ini extension, these files are not true ini files, since they allow nesting and conditional logic on values.

    Expectation data

    Because web-platform-tests are intended to check the standard behavior rather than the behavior of specific implementations, it is common for tests to fail in specific implementations. Therefore, the expected result of each test that doesn't pass is kept in the corresponding metadata file. A simple example of such a file might be:

    [filename.html]
    type: testharness

    [Subtest name for failing test]
        expected: FAIL

    [Subtest name for erroring test]
        expected: ERROR

    Expectations can be made platform-specific using a simple set of python-like conditions. For example, for a test that times out on Linux and fails on other platforms:

    [filename.html]
   type: reftest
    expected:
        if os == "linux": TIMEOUT
        FAIL

    The available set of condition variables is that provided by mozinfo plus a boolean variable e10s that is true when e10s is enabled.

    Expectation data can be updated automatically on the basis of a test run (e.g. from try), using the raw structured log files. On treeherder, these are files listed as artifact uploaded: wpt_raw.log. Locally they can be generated using a mach command like:

    ./mach web-platform-tests testing/web-platform/tests/dom/historical.html --log-raw=historical.log

    Then to update the expectation data:

    ./mach web-platform-tests-update historical.log

    By default, this will make a commit (in a git tree) or an mq patch (in an hg tree). To override this behavior, use --no-patch. If the expected result of a test is platform-specific, you must provide multiple logs—one for each platform—to the update command, or the correct metadata will not be generated. If you want to wipe away the existing metadata for the test rather than try to update the conditions in place (such as when a test goes from having platform-specific behavior to generic behavior) use --ignore-existing.

    Disabling tests

    You can disable a test by adding a disabled key to the metadata file. The value can be anything except the special values @False and @Reset, but by convention is the bug number detailing the reason the test was disabled. For example:

    [filename.html]
   type: testharness
    disabled:
        if os == "win": https://bugzilla.mozilla.org/show_bug.cgi?id=1234567

    Setting prefs

    Per-test prefs can be set using a metadata item called prefs which takes a list of preferences to set and the values to assign to them. This lets tests run against features which are preferenced off by default, for instance.

    [filename.html]
    prefs: [dom.serviceWorkers.enabled:true,
            dom.serviceWorkers.interception.enabled:true,
            dom.serviceWorkers.exemptFromPerDomainMax:true,
            dom.caches.enabled:true]

    Per-directory settings

    A __dir__.ini file creates metadata that applies to the entire subtree beneath it. For example, we can disable all DOM tests using a file in testing/web-platform/meta/dom/__dir__.ini file:

    disabled: https://some-bug

    Having done this, we might need to re-enable a specific test under dom/. This can be done using the special value @False in the metadata file for that test:

    [filename.html]
    disabled: @False

    With lists of preferences, the preferences applied are typically those for the test and those from any __dir__.ini files up to the test root. If we need to apply a different set of prefs to a specific test, the special value @Reset prevents all inheritance of prefs from further up the tree.

    Running tests in other browsers

    Servo

    Servo has its own import of web-platform-tests, with tests living under tests/wpt/web-platform-tests/. It also has mach commands similar to those provided in the Gecko build system, with slightly different names. In particular, to run the in-tree copy one can do this:

    ./mach test-wpt

    in a Servo checkout.

    Chrome

    While it's possible to run tests against Google Chrome, it's currently a little tricky.

    1. Download the chromedriver binary and place it somewhere, such as ~/bin/chromedriver.
    2. Activate the virtualenv in your object directory (such as by executing source objdir/_virtualenv/bin/activate) and install the extra Python requirements for running in Chrome: 
      pip install -r testing/web-platform/harness/requirements_chrome.txt
    3. Copy testing/web-platform/meta/MANIFEST.json somewhere, such as ~/chrome-metadata. This is needed because the Gecko metadata files are not compatible with Chrome.
    4. Copy testing/web-platform/wptrunner.ini to testing/web-platform/wptrunner.chrome.ini and edit it to look like this: 
      [products]
chrome =

[web-platform-tests]
remote_url = https://github.com/w3c/web-platform-tests.git
branch = master
sync_path = sync

[paths]
prefs = ../profiles
run_info = .

[manifest:upstream]
tests = tests
metadata = /home/username/develop/chrome-metadata/
url_base = /
    5. Run the tests: 
      cd testing/web-platform
python runtests.py --product chrome --config wptrunner.chrome.ini --webdriver-binary ~/bin/chromedriver --log-mach -

    Document Tags and Contributors

    Tags: 
     Contributors to this page: rolfedh, Jonathan_Watt, Sheppy, jgraham
     Last updated by: rolfedh,