Firefox OS mochitests

Running mochitests on B2G desktop builds

Before running B2G on B2G desktop, you'll need to install Gaia and build a profile.

Using mach

Mach is a command-line interface to help developers perform common tasks. Let's look at how to use it run mochitests. If you have a local B2G Desktop build with marionette enabled (specify ENABLE_MARIONETTE=1 in your mozconfig), you can use mach to run all of the mochitests. Run the following command from the root of your B2G source tree:

./mach mochitest --profile path/to/non/debug/gaia/profile

Alternatively you can set the $GAIA_PROFILE environment variable to avoid having to pass it in all the time.

To run a specific test, you can run the following:

export GAIA_PROFILE=path/to/non/debug/gaia/profile
./mach mochitest test_path

To see mochitest options:

./mach mochitest --help

Aside from requiring a gaia profile, b2g desktop mochitests support the same options and are run in the same way as desktop mochitests. See the Firefox mochitest documentation for more information on running tests.

Using the mochitest runner

You can run mochitest-plain on B2G Desktop builds.

Prerequisites

• You need to be using a Marionette-enabled build (compiled with ENABLE_MARIONETTE=1 in mozconfig).
• You need to have manually generated a gaia profile (which you should already have if you've been running a B2G desktop build manually); this must be a non-DEBUG profile.
• You cannot use a Gaia profile that was made with a custom GAIA_DOMAIN.

Setup for mozilla-b2g18 trees

cd $GECKO_OBJ_DIR
python $GECKO_SRC_DIR/python/virtualenv/virtualenv.py venv
source venv/bin/activate
cd $GECKO_SRC_DIR/testing/marionette/client
python setup.py develop

Setup for trunk trees, including mozilla-central

cd $GECKO_OBJ_DIR
source _virtualenv/bin/activate

Running the tests

You can then run mochitests using the following commands:

cd $GECKO_OBJ_DIR/_tests/testing/mochitest
python runtestsb2g.py --desktop --console-level INFO --profile /path/to/gaia/profile --app /path/to/b2g

The test runner will launch your b2g desktop build and run the tests: all B2G mochitests by default, which is very time consuming. Other common options are as follows:

• To run specific tests, pass in --test-path.
• You can optionally add --total-chunks and --this-chunks arguments as you would with regular desktop mochitest.
• Finally, you can use python runtestsb2g.py --help for a full list of supported arguments.

Running mochitests on emulator builds

You can run mochitest-plain on B2G device builds. Currently, these work best when run on an emulator.

Using mach

If you have a local emulator build that is made against a trunk Gecko branch (not mozilla-b2g18), the easiest way to run mochitests on it is with mach. To run all of the tests, do the following:

cd $B2G_DIR
./mach mochitest

To run a specific test, use this:

./mach mochitest test_path

To see mochitest options, you can use this:

./mach mochitest --help

B2G emulator and device builds support the same options and are run in the same way as desktop mochitests. See the Firefox mochitest documentation for more information on running tests.

Using the mochitest runner

Similar to before, you can also run mochitest on device or emulator builds of B2G using the mochitest runner. The prerequisites are a little more complex.

Pre-requisites

• You need to build B2G for the target you're testing (see Building and installing Boot to Gecko).
• You cannot use a Gaia build that was made with a custom GAIA_DOMAIN.
• You need to have a desktop version of xpcshell (it needs to run on the host); you will have this already if you have a copy of Firefox on your system.
• You need to install some Python packages that are needed, either in a virtualenv or otherwise. Read on for more details.

Setup for trunk trees, including mozilla-central

cd $GECKO_OBJ_DIR
source _virtualenv/bin/activate

Running the tests

You can then run mochitest-plain using one of the following sets of terminal commands.

To run tests on a device, both the device and the host computer must be on the same wireless network because the device will request files from an HTTP server that's started by mochitest on the host computer. To start the tests, use the following:

adb forward tcp:2828 tcp:2828
cd $GECKO_OBJ_DIR/_tests/testing/mochitest
python runtestsb2g.py --b2gpath $B2G_HOME --xre-path /path/to/dir/containing/desktop/xpcshell --console-level INFO \
  --httpd-path ./

If you're using an emulator, use the following:

cd $GECKO_OBJ_DIR/_tests/testing/mochitest
python runtestsb2g.py --b2gpath $B2G_HOME --xre-path /path/to/dir/containing/desktop/xpcshell --console-level INFO \
  --httpd-path ./ --emulator arm

The above commands will run all B2G mochitests by default, which is very time consuming. Other common options are as follows:

• To run specific tests, pass in --test-path.
• You can optionally add --total-chunks and --this-chunks arguments as you would with regular desktop mochitest.
• You can add --chrome to run mochitest chrome tests instead of mochitest-plain. See Chrome tests for more information on those.
• Finally, you can use python runtestsb2g.py --help for a full list of supported arguments.

After you invoke runtestsb2g.py, the test runner will launch the emulator for you (if you're running the tests on an emulator) or reboot your device (if you're running the tests on a device), and start running the tests. Because the emulator is slow, and it is necessary to push a test profile to the emulator and restart B2G, the tests can take a few minutes to start. Before they start, you will just see a black or white screen. After they start, you should see the test log being dumped to the console.

When the tests are done, the emulator is shut down for you, or if you're using a device, the device is rebooted.

Running mochitest with a downloaded emulator

If you've built B2G for another config (like otoro) and want to run the tests on an emulator, you can do so without building an emulator yourself. Just download the latest trunk arm emulator, and use the same instructions as above, replacing the --b2gpath $B2G_HOME argument for runtestsb2g.py with --b2gpath /path/to/unpacked/emulator.

Running mochitest with a downloaded emulator and downloaded tests

You can also run mochitests on an emulator without building or cloning anything. To do this, you need to download the latest trunk arm emulator and unpack it, and download the latest tests.zip file (from the same place as the emulator) and unpack it. 

You'll also need virtualenv installed on your system.

Then you can set up your environment and run the tests:

virtualenv venv
source venv/bin/activate
cd $TESTS_DIR/marionette
python setup.py develop
cd $TESTS_DIR/mochitests 
python runtestsb2g.py --b2gpath /path/to/extracted/emulator --xre-path /path/to/desktop/xpcshell --console-level INFO \
  --httpd-path ./ --emulator arm

Technical Implementation Details

There are a few things that happen when you run B2G mochitests.

1. The homescreen system app is replaced by a special certified app called test-container (note this is checked in to gaia). In theory this test-container is meant to be generic such that any test harness can use it, but so far only mochitests do.
2. The test-container app exposes a single <iframe mozbrowser mozapp> in which the mochitests are loaded (mochitests have their own manifest.webapp).
3. The automation replaces the homescreen app with the test-container app using a pref.
4. The automation loads the mochitest app into the iframe the test-container app exposes (this script is executed with chrome privileges by marionette).
5. The automation loads special powers into the child frame that is now running mochitests and sets up the message manager (this lets mochitests access some privileged apis).
6. The automation loads the mochitest url into the mochitest app's iframe. This kicks off the tests.