Endurance tests

The Gaia-UI Endurance tests are a suite of Firefox OS tests built on top of the Gaiatest (Gaia UI Tests) Framework. The tests run on real Firefox OS devices, and use Marionette to drive the device's UI.

As mentioned above, the Gaia-UI Endurance Tests verify Firefox OS device and Gaia application stability by performing several iterations of common use cases. Examples of such use cases include adding a contact to the address book, adding an event to the calendar, sending an SMS message, opening the Firefox browser and browsing to a specific site.

As well as verifying Gaia application functionality under numerous iterations, the endurance tests also perform device-side "checkpoints" after every N number of iterations. This checkpoint currently consists of recording the device’s B2G process RSS memory use value. By logging these results over time and comparing them across commits and builds, we can to detect any sudden changes in B2G memory consumption that may occur during the various test scenarios.

Running the tests

Let's go through the steps required to set up the Gaia-UI Endurance test environment and run the tests on your local machine and Firefox OS device.

Prerequisites

• Ubuntu 12.04 (or better) x64, or Mac OS X (these instructions are confirmed for 10.8 but should work on previous versions of 10, theoretically.)
• A Firefox OS device ALREADY FLASHED with an ENGINEERING build of Firefox OS B2G. As the Endurance tests are underpinned by the well maintained gaiatest, compatibility with a range of devices and branches is good.
• ADB installed, and the environment variable ADB_PATH pointing to your ADB location.

If you are on Ubuntu, you need to check that it is configured to support the USB connection to the Firefox OS device. To verify this, connect the device to your computer via USB, open a terminal and enter the adb logcat command to see if it connects.

Step 1: Clone the Gaia repository

The Gaia-UI Endurance Tests are located in the Mozilla Github Gaia repository. Assuming that you haven’t done so already, the first step is to clone that repo:

git clone https://github.com/mozilla-b2g/gaia.git

You may want to go grab a coffee and come back in five minutes.

Step 2: Run the GaiaTest setup

The Gaia-UI Endurance tests are built upon the GaiaTest framework (which uses Marionette). The next step is to run the setup script to install GaiaTest and all of the required dependencies. You may wish to create a new virtual environment to use with the Gaia-UI Endurance Tests. If you don’t, you may need to use sudo while running the setup command. In your terminal, type:

cd gaia/tests/python/gaia-ui-tests
python setup.py develop

Step 3: Set test vars and acknowledge risks

GaiaTest uses a special file to set certain variables that are required for test configuration. For example, to tell the device which WiFi network it should use. Before running the Gaia-UI Endurance Tests, you must setup the test vars file. Make a copy of the gaia/tests/python/gaia-ui-tests/gaiatest/testvars_template.json file in the same location (call it what you like) and edit it:

• If the Firefox OS device has a SIM card, add the corresponding phone number in the phone number entry in the carrier section, e.g. "phone_number": "5551234567".
• Add the same phone number for the "remote_phone_number" entry.
• In the wifi section, add the SSID for the Wifi network that the Firefox OS device is to connect to during the tests in the ssid line, for example "ssid": "Wifi Network Name".

As mentioned in the warning above, running the Gaia-UI Endurance tests will result in data being erased from the Firefox OS device and microSD card. This is to ensure that the tests start cleanly each time. For example, running a test on a device that already has 10,000 contacts will have very different memory value results vs running the same test on a device with no existing contacts. In order to run the tests, you must acknowledge that you are aware of this data loss risk.

To acknowledge the risks, add the following entry to your testvars file as the first entry in the list: "acknowledged_risks": true.

Step 4: Connect the device to WiFi

Next, power on your Firefox OS device. After it boots up, go into the Wifi settings then manually select and connect to the Wifi network that you want the device to use for the tests. Ensure the Wifi settings confirm that the device is connected to Wifi before continuing.

Step 5: Connect to USB and ADB Forward the Device

Attach the Firefox OS device to your computer via USB.

Now tell adb to forward the device port to GaiaTest using the following command:

adb forward tcp:2828 tcp:2828

adb root
adb forward tcp:2828 tcp:2828

Step 6: Run a Test

Now you’re ready to actually try running a test. To run the add_contact endurance test, with a single iteration, use the following commands:

cd gaia/tests/python/gaia-ui-tests
gaiatest --type=b2g --address=localhost:2828 --testvars=mytestvars.json --iterations=1 --checkpoint=1 --restart gaiatest/tests/endurance/test_endurance_add_contact.py

If you get a “connection refused” error it means the device USB connection didn’t work; just repeat the device connection and try again.

The Firefox OS device b2g process should now restart, then the add_contact endurance test will run with a single iteration. If you watch the Firefox OS device, you’ll see the device UI being manipulated by Marionette. After the test finishes, a memory checkpoint will be performed.

The test result will be displayed in the terminal window. Note that this result doesn’t include the b2g process memory value; this value is stored in a text file that was created at the time of the checkpoint in the checkpoints directory. To see the resulting b2g process, open this file. This "suite_summary" file will contain the average b2g process memory use (RSS) value, averaged from all of the test checkpoints (in our example there was only one checkpoint anyway).

There are two other files present in the checkpoints folder:

• The checkpoint_add_contact_(datetime)_summary.log file contains a summary for the test run. This includes the number of iterations, all of the RSS value readings from each checkpoint, the average RSS value, etc.
• The checkpoint_add_contact_(datetime).log file contains the raw data received from each of the device checkpoints, from which the summary files were produced.

Command Line Options

Here is a description of the GaiaTest command line options that you can use when running the Gaia-UI Endurance Test(s):

• --type: The type of test being run; always use --type=b2g for this.
• --address: The address where the device port is forwarded, when you issued the ADB forward command; just use --address=localhost:2828.
• --testvars: The path and name of the file containing the test vars, as described in Step 3 above.
• --iterations: The number of times to repeat each test case/test; i.e. 100 iterations of the add_contact test will result in 100 new address book entries being added.
• --checkpoints: After every N iterations a b2g process memory checkpoint will be grabbed from the device; by default a checkpoint will be performed after all iterations are complete.
• --restart: The Firefox OS device’s b2g process will be restarted before each new test (not each iteration) begins (i.e. a soft reset.)

As an example, to run the add_contact test and have it perform 100 test case iterations, grabbing a device memory checkpoint after every 10 iterations, and restarting the b2g process before the test begins, you would use the following command line:

gaiatest --type=b2g --address=localhost:2828 --testvars=mytestvars.json --iterations=100 --checkpoint=10 --restart gaiatest/tests/endurance/test_endurance_add_contact.py

If you wish to run the entire Gaia-UI Endurance test suite, instead of specifying a single test on the command line, just point to the manifest file for the test name, like this:

gaiatest --type=b2g --address=localhost:2828 --testvars=mytestvars.json --iterations=1 --checkpoint=1 --restart gaiatest/tests/endurance/manifest.ini

However, note that running the entire test suite, with 100 iterations of each test (and a checkpoint after every 10 iterations) takes approximately 16 hours.

Now you should be all set and have the Gaia-UI Endurance Tests up and running on your local machine/device.

Contributing to the project

If you have any questions about the Firefox OS Endurance tests or are interested in contributing to this important automation development effort, feel free to contact us in the Mozilla #automation IRC channel.