Running tests on Firefox OS: A guide for developers

Getting started

If you are a Gecko developer, then you should review the Firefox OS-specific documentation for the test automation you are already familar with: mochitest, reftest, and xpcshell.

If you are a Gaia or App developer, or if you're a Gecko developer interested in doing a deeper "end user" style test where you actually exercise the OS, then you'll need to look into the Gaia test suites. There are two primary test suites:

• Gaia UI Tests:  These tests are Python-based, and can be run on both desktop and devices. These tests test various aspect of the Gaia UI such as endurance tests (adding lots of contacts to the Contacts app, unlocking the lock screen repeatedly, setting many alarms in the Alarms app, etc.), functional tests (e.g. finding stations on the FM radio app, capturing a photo using the Camera app) and accessibility tests (e.g. enabling and disabiling the screenreader.)
• Gaia Integration tests: These tests are JavaScript-based, using marionette-js-runner, and can only be run on B2G desktop builds at the time of this writing (real device support is coming very soon.) These test various aspects of how Gaia interacts with other apps and services, so for example, if the Calendar app can successfully add a CalDAV server, or if the Browser app can successfully handle interactions with search engines.

Which one you choose really depends on your preferred toolchain, and what things you want to test.

Let's now move and look at running these tests.

Running the Gaia UI Tests

The Gaia UI Test suite can be run on real devices and B2G Desktop builds, but in this section we'll concentrate on running them on a real device, as real devices are always best where possible.

Note that this test is destructive and as such, you should back up anything you care about on the phone before running these tests. Depending on which tests you run, they can also make phone calls. So be aware that you want to be very careful about what you run and how you back up the phone, remove the SIM card, etc.  That said, if you've already created an engineering build they are really easy to get running. Here's how.

One Time Set up

You only need to perform the following steps once, assuming you do not change the location of your Gaia directory. Create a python virtualenv (install the virtualenv tool first if you haven't already), activate it, and install the gaia UI test tool into your virtualenv. By creating the virtual environment using the steps below, you ensure that you are running the gaia UI test harness code that lives in your Gaia repo (that's useful in case you need to debug anything).

$ virtualenv gaia_ui_venv # This will create a gaia_ui_venv directory where the virtual environment lives. It can be anywhere on your system.
$ source gaia_ui_venv/bin/activate # This activates our virtualenv
(gaia_ui_venv)$ cd <b2groot>/gaia/tests/python/gaia-ui-tests;python setup.py develop # This installs the gaia ui harness into your virtual environment. 

If you have already created a virtual environment for gaia ui tests, you can simply do the following:

$ source gaia_ui_venv/bin/activate

To Run the Tests

First you need to create the testvars file. To do this, copy the standard one over, and add in the attributes to turn off the warnings that this test will destroy all content on your phone. These are good tests, they leave no state around and as such, you will need to make sure your phone is backed up before running them. Instructions from here on will assume you've activated the virtual environment and are working in the gaia/tests/python/gaia-ui-tests directory.

(gaia_ui_venv)$ cp gaiatest/testvars_template.json testvars.json
# Now edit your copy of testvars.json and add in the following attributes into the json:
"acknowledged_risks": true,
"skip_warning": true,

Now you just need to connect our phone via USB, forward the marionette port so your test runner can access it and run our tests. The tests are in gaiatest/tests and you can pick whichever one you want to run. For example, if you wanted to run contacts tests you would do the following:

(gaia_ui_venv)$ adb forward tcp:2828 tcp:2828
(gaia_ui_venv)$ gaiatest --testvars=testvars.json --address=localhost:2828 gaiatest/tests/functional/contacts/

To get out of the python virtualenv, just use the special virtualenv command deactivate:

(gaia_ui_venv)$ deactivate
$

Running the Gaia Integration tests

To run the Gaia Integration tests tests you currently have to use a B2G Desktop build (note that these are also going to be available for devices soon as well). Let's look at how this is done.

These just require a Gaia tree and NodeJS to be installed on your computer; the following command will do the rest:

$ cd gaia $ make test-integration 

That's it — this instruction will download a B2G desktop build, and start running the tests in that build.

Wrapping Up

As always, work is underway to make all our tests easier to run both locally for developers as well as in our automation systems. Feel free to drop into the #ateam channel any time you have questions about test automation for Firefox OS or any of the Mozilla automation tools.