Gaia UI Tests: Running Tests

The gaia-ui-tests have been developed with the capability to run on a real device device, emulators, and the B2G Desktop client. By far the easiest method is to run against B2G Desktop. Please note that you should choose the right branch of Gaia to run against the version of Firefox OS that you are using.

Testing on B2G Desktop

The B2G desktop client is as it sounds: it's a desktop version of B2G that you can use to perform web app tests on Firefox OS from the comfort of your desktop. If you are not familiar with it, you can find more out by reading Using the B2G desktop client. This section will show you how to run gaia-ui-tests on B2G desktop.

The desktop B2G client is ideal for developing tests when a real device is unavailable and there is no need for phone-level functions. It is fast, accessible, and will run on Linux, Mac, and Windows!

Running the tests

We'll look at running new tests; we'll install the in-tree version of gaiatest directly from the Gaia repository so that our changes are picked up immediately. In your cloned version of Gaia, navigate to gaia/tests/python/gaia-ui-tests and run the setup command:

python setup.py develop

This might give permission denied errors when not running it inside a virtual environment. In that case, install virtualenv and virtualenvwrapper, if you haven't already done so:

pip install virtualenv
pip install virtualenvwrapper (perhaps need to use sudo here)
export WORKON_HOME=~/Envs
source /usr/local/bin/virtualenvwrapper.sh
echo "source /usr/local/bin/virtualenvwrapper.sh" >> .bash (rc or _profile)

Then, create and enter the virtual environment:

mkvirtualenv gaia-ui-test

Via Marionette, gaiatest is able to launch the B2G Desktop when we start the test. To do this, you need to pass in the path to the B2G binary file and B2G profile. In the below example we have included the $HOME variable as the path to the b2g binary and the b2g profile. This will simply be the location that you unzipped B2G Desktop to after you downloaded it. The command to run all the tests is:

gaiatest --binary=$HOME/b2g/b2g-bin --profile=$HOME/b2g/gaia/profile --testvars=testvars.json --restart --type=b2g ./gaiatest/tests/functional/manifest.ini

Tests that are incompatible with the B2G Desktop client — for example those that make calls and send SMS — are automatically skipped from the test run.

You can run a test directly by calling the file on the command line:

gaiatest --binary=$HOME/b2g/b2g-bin --profile=$HOME/b2g/gaia/profile --testvars=testvars.json --restart ./gaiatest/tests/functional/clock/test_clock_set_alarm.py

Testing on Firefox OS devices

Testing on real Firefox OS devices is more complicated, but for obvious reasons gives the most accurate results. With access to all available device APIs, hardware features, and so forth. However, you can test on the emulator if you don't have a real Firefox OS device, or if the features you're using don't require real hardware. You can also test general functionality in the emulator.

Prerequisites

• Android Debug Bridge installed and udev rules configured.
• A Firefox OS device with a Marionette-enabled build.
• The Gaia Github repository branch that matches your Firefox OS device cloned and checked out.
• A testvars.json file; see Configuring test variables.

Running the tests

The gaiatest package is needed to run the tests. We need to install this for the branch that we want to run the tests on. Navigate to the gaia/tests/python/gaia-ui-tests/ directory and install gaiatest with this command:

python setup.py develop

Marionette on the device awaits commands on port 2828. We need to forward our local port to the remote port on the device using adb. Run the following command:

adb forward tcp:2828 tcp:2828

Please be reminded that you'll need to start your emulator before the command, if you are not running on a real Firefox OS device.

You can use the following command to execute all the gaia-ui-tests:

gaiatest --address=localhost:2828 --testvars=<TESTVARS_FILE> --restart --type=b2g gaiatest/tests/functional/manifest.ini

Or use the following if you want to run a single test (substituting the last part for the actual test you want to run):

gaiatest --address=localhost:2828 --testvars=<TESTVARS_FILE> --restart gaiatest/tests/functional/clock/test_clock_create_new_alarm.py

After the test, you will see all the pass/fail results and stacktraces for each file.

Testing locally with Treeherder configuration for Gecko and Gaia developers

In this section we will focus upon replicating a Treeherder test locally. Before testing locally with a Treeherder configuration, you need to be aware that it builds a Gaia profile separately. The profile that comes packaged with the Treeherder build or rel-eng build is NOT the same Gaia profile that Treeherder uses.

Building a Gaia profile with Treeherder's  settings

Before building a profile we need to check the environment variables that were used to build the profile. Searching in the "brief log" of the Gu test run, find this command block:

Using env: {'DEBUG': '0',
            'DESKTOP': '0',
            'DESKTOP_SHIMS': '0',
            'NOFTU': '0'}

These settings can drastically change the way desktop B2G behaves. It is very important to match Treeherder's settings.

The next step is to checkout the git commit that matches the hg commit that Treeherder ran with. You can also find the hg commit in the "brief log" of the Treeherder run but you will need to correlate that with the git commit yourself.

The gaia test repository includes a make script for building a Gaia profile. Navigate to the Gaia repo that you have cloned locally, and build the profile using the environment variables we found on Treeherder:

DEBUG=0 DESKTOP=0 DESKTOP_SHIMS=0 NOFTU=0 make

Running the tests

Using the same git commit that Treeherder used to run the test, go to the gaia/tests/python/gaia-ui-tests directory and install gaiatest using the following command:

python setup.py develop

Now that you have built the Gaia profile, the only significant difference to running these tests is that you must direct gaiatest to the binary location and profile location.

Inside the gaia-ui-tests directory, run the below command, with the following substitutions made:

• $BINARY_PATH: The path to the b2g-bin file in the directory you have built or unpacked desktop B2G to.
• $PROFILE_PATH: The path to your locally build Gaia profile. If built from the Gaia repository it will be ~/gaia/profile.

gaiatest --binary $BINARY_PATH/b2g/b2g-bin --profile $PROFILE_PATH/profile --restart --testvars testvars.json gaiatest/tests/manifest.ini --type=b2g

Finding the HTML report for a completed Treeherder job

All Treeherder jobs (both pass and fail) have an HTML report output generated and stored.

1. For the commit and Gip job, click the chunk of interest (e.g., f1, f2).
2. In the status bar at the bottom of the screen find the section titled: artifact uploaded output.html.
3. Click link to view the report.

Testing against an already running Desktop B2G binary

In this section we will look at running the tests against an already running Desktop B2G binary with modified source code.

Running the tests

Running tests against an existing b2g process is quite easy — you've already gone through the hard parts to build B2G!

If you have not already installed gaiatest then do so now by going to gaia/tests/python/gaia-ui-tests and running the following:

python setup.py develop

Once you have started the b2g binary process and attached your debugger you need to direct gaiatest to the port using the --address command. Do not use --binary or --profile, otherwise gaiatest will just send commands directly to the port without attempting to start or close the binary.

gaiatest --testvars testvars.json gaiatest/tests/functional/test_that_is_being_debugged.py

Configuring test variables

You can customize variables used during the test using a testvars.json file. The gaia-ui-tests will wipe the databases on your phone in order to give the test a clean profile to run against. Gaiatest contains protection against running and wiping your device's data. Please read the warning before you set up your test variables file.

We use the --testvars option to pass in local variables, particularly those that cannot be checked into the repository. For example in gaia-ui-tests these variables can be your private login credentials, phone number, or details of your WiFi connection.

To use it, copy testvars_template.json to a different filename but add it into .gitignore so you don't check it into your repository.

When running your tests add the argument: --testvars=(filename).json

How to configure WiFi using testvars.json (for devices)

By setting the WiFi key in testvars.json you can configure the Firefox OS Settings database with your WiFi configuration. Then the test cases can use your WiFi connection.

No WiFi authentication:

"wifi": { "ssid": "MyNetwork"} 

WEP authentication:

"wifi": { "ssid": "MyNetwork", "keyManagement": "WEP", "wep": "MyPassword" } 

WPA-PSK authentication:

"wifi": { "ssid": "MyNetwork", "keyManagement": "WPA-PSK", "psk": "MyPassword" } 

Troubleshooting

This section details some common error messages, offering potential solutions for each.

Common problems before the test has started