Testing Gaia code changes

When you have finished making a change to the Gaia codebase and it seems to work ok, the next step is to go through a testing procedure to make sure your changes really work — and are working ok with the rest of Gaia — before submitting a patch back to the project. This article explains how.

The testing procedure generally consists of:

Standard debugging procedure
Running automated tests

Let's examine both of these areas now.

Standard debugging

If your are an experienced web developer then debugging Gaia code should be a familiar process. We already discussed how to run Gaia in your Desktop Firefox installation, and how to make a basic change. For more complex code base additions, you'll want to make a lot more use of the Firefox debugging tools accessible in Desktop Firefox view.

Note: More instructions for using all these tools can be found in our Tools zone.

Automated tests

You should also run the standard automated test suites that code with Gaia before submitting a patch, to make sure that your code changes do not adversely affect any of the essential existing phone functionality. The tests you can run are:

unit tests
integration tests
performance tests
UI tests

We generally ask that you run the tests before submitting a patch; if it is your first contribution then you can submit without tests, but you need to ask for help with running the tests in future. You should update the Gaia repo before running tests, to make sure you have the latest ones.

Shortcut: If you don't know how to setup the testing environment, try MozITP — this toolset automatically sets up an Ubuntu VM and testing environment allowing you to run Integation tests and UI tests with little more than a single click.

Note: You can find more information on how these tests work at the Firefox OS Automated testing page.

Note: You should consider running each set of tests on a real device if available (some features/hardware aren't supported on an emulator), or on the B2G Desktop emulator or Firefox Nightly if not.

Unit tests

Unit tests are tests on individual units of code in a larger application — in the case of Gaia, individual apps. Gaia uses:

mocha for its test framework
chai as an assert library
sinon.js for a mock & stub library
blanket.js as a test coverage tool

You can run the following command to download, install and host a unittest server (running it takes a good few minutes, so this might be a good time to make a cup of tea):

DEBUG=1 make
export FIREFOX=/Applications/FirefoxNightly.app/Contents/MacOS/firefox
bin/gaia-test

This will open a local web page with a list of unit tests. To run them:

Select the tests you want to run from the list in the page (an asterisk appears beside them).
Click the "Execute" button.
Scroll to the bottom of the page to see the results.

With the window open, you can also run the full test suite from a new terminal window:

make test-agent-test

Note: This can take quite a long time, as there are a lot of tests to run (possibly an hour or more), therefore you’ll probably just want to run the tests for the app you’ve modified. You can do this by appending APP=<app folder name> onto the command, for example APP=settings.

Note: You can also read Gaia Unit Tests for more unit test information.

Integration tests

Integration testing involves testing different units of code together in a group to see how well they work together, and is the logical next step after unit testing. Gaia Integration tests are driven by a marionette script written in JavaScript and a python-based server. It can communicate with Gecko so it’s possible to control both the browser and Firefox OS device, and get them interacting with each other.

You can run the following command to trigger the integration tests:

make test-integration

Note: As with the unit tests, running the whole integration test suite can be very time consuming, so you can append APP=<app folder name> onto the above command to test a single app, for example APP=calendar.

Note: If you want to run integration tests on different (non-phone) device types, you should append GAIA_DEVICE_TYPE=<device type name> onto the above command to signify the device type, for example GAIA_DEVICE_TYPE=tv. If you don't do this, the tests may fail.

Note: for more on integration tests, read Gaia integration tests.

Performance tests

Gaia performance tests are run with the external Raptor CLI tool. With Raptor you can automate launching an application several times, and it will output statistics about the app's launch time. After running a test, Raptor also displays memory information about the application.

To run Raptor's performance tests, please visit Raptor: Getting Started.

UI tests

See Gaia UI Tests Introduction.