Release Testing

To shorten the times for QA while running tests against upcoming Firefox releases, Mozmill is used to run existing automated tests. The remaining tests still have to be run manually but their number is growing smaller over time. Test results are reported to Brasstacks, where they can be analyzed and synchronized with Litmus later.

The tests which are run for release testing of Firefox are:

• Smoketests, BFT's, and parts of FFT's against release candidate builds
• Software Update tests against the latest release and the first release of the same branch

In the following sections, both procedures of testing will be described in detail.

Smoketests and BFT test-run

For new Firefox release candidate builds, QA is running Smoketests and BFT's against that build. For an example of this, see the results of the Firefox 3.5.8 test plan. To speed-up the test-run, Mozmill is used to run all existing automated tests from the Smoketests and BFT testgroups in Litmus. These automated tests are run on each of the supported platforms. The results are then reported to Brasstacks directly and then manually synchronized with Litmus.

The automated test-run is separated into two independent Mozmill test-runs. The first of these is for normal Mozmill tests and the second run runs tests which require a browser restart. Both runs are started with the command line Mozmill client (see here for installation details).

Preparation

Follow the notes in the test plan and obtain the release candidate builds from the FTP server. Store them locally and install the builds within the operating systems to be tested. To start a complete test-run you should use the firefox-automation scripts which are located in the Mozmill repository on Github. Clone that repository or simply download the latest version of the master branch to your local disk.

Starting the test-run

To start the test-run for all kinds of BFT tests, run the command given below:

testrun_general.py --report=https://mozmill-release.brasstacks.mozilla.com/db/ /Applications/Firefox.app
testrun_general.py --report=https://mozmill-release.brasstacks.mozilla.com/db/ /home/mozilla/Desktop/firefox/firefox
testrun_general.py --report=https://mozmill-release.brasstacks.mozilla.com/db/ /c/firefox/firefox.exe

QA lab machine

There is a machine in the QA lab which runs daily automated tests and has a pre-configured environment for the automation. This machine is also used for release testing. To run all the Mozmill tests, do the following steps:

1. Download all builds to '/data/testing/%version%/bft_%platform%' within the master OS X system.
2. Open a shell and execute the following commands:

$ cd /data/testing/               (Mac OS X)
$ cd /z/data/testing/             (Windows)
$ cd /mnt/hgfs/data/testing/      (Linux)

./release_bft.sh %branch% %platform% %build%

Script Variables

%branch% - version of Firefox you are testing (Example: 3.5, 3.6, 3.7, ...)
%platform% - platform you are testing on (Example: mac, win, linux) which is just a part of the folder name under %branch%
%build% - (optional) only test the given build under the folder specified with %platform% 

Reports

All reports are sent to Brasstacks. Open the summary page and scroll down to the bottom of the page to find the latest results. Link those results from the test plan results page.

Synchronizing results with Litmus

To have all the results from Brasstacks available in Litmus a manual synchronization has to be run. Therefore log into Litmus and start the BFT test-run specified by the test plan for the release build to test. After selecting the operating system choose the "Mozmill Tests" testgroup and "Mozmill - All" subgroup. At the following page you will have to mark all results as passed first. Use the following bookmarklet which will set the checkboxes of all tests to the PASS state. Afterward check the Brasstacks results (normal and restart test-run) and mark a test as not-run if it is failing for the automation.

To create the bookmarklet add a new bookmark to e.g. your bookmarks toolbar and enter the following code as the location:

javascript:(function(){for%20each%20(input%20in%20document.getElementsByTagName('input'))%20if%20(input.type%20==%20"radio"%20&&%20input.id.indexOf("pass")%20!=%20-1)%20input.checked%20=%20true;})()

Software Update tests

For new Firefox releases, QA tests all possible update paths to make sure users on the beta and release channels will receive the right updates. Automated Software Update tests are run on selected platforms. During these test-runs, the results are displayed within the shell or can be written to a log file.

The Software Update tests are separated into two independent test-runs for each build. The first of these checks and updates Firefox in the normal way and the second run updates with the fallback case. The Software Update tests use the command line Mozmilll client (see here for installation details).

Preparation

Follow the notes in the test plan and download the specified Firefox releases from the FTP server to a local folder within the operating system being tested, e.g. 'd:/builds/'. To start a complete test-run you should use the firefox-automation scripts which are located in the Mozmill repository on Github. Clone that repository or simply download the latest version of the master branch to your local disk.

Test-run

During the release test cycle, tests will be run with the betatest, beta, releasetest, and release channels. For each of those channels the following command is executed:

$ testrun_update.py --report=https://mozmill-release.brasstacks.mozilla.com/db/ --channel=%channel% (build|installer|folder)

Examples:

$ testrun_update.py --report=https://mozmill-release.brasstacks.mozilla.com/db/ --channel=beta /d/builds
$ testrun_update.py --report=https://mozmill-release.brasstacks.mozilla.com/db/ --channel=beta /Users/john/Desktop/builds/
$ testrun_update.py --report=https://mozmill-release.brasstacks.mozilla.com/db/ --channel=beta /home/john/Desktop/builds

QA Lab Machine

The machine in the QA lab which runs our daily automated tests will also be used for software update tests. It has a pre-configured environment for testing. To run the tests, follow these steps:

1. Download all the builds to '/data/testing/%branch%/update_%os%'.
2. Open a shell and execute the following commands with the operating system to be tested:

$ cd /data/testing/               (Mac OS X)
$ cd /z/data/testing/             (Windows)
$ cd /mnt/hgfs/data/testing/      (Linux)

$ ./release_update.sh %branch% %os% %channel% %build%

Script Variables

%branch% - version of Firefox you are testing (Example: 3.5, 3.6, 3.7)
%os% - platform you are testing on (Example: mac, winxp, win7, linux) which is just part of the folder name under %branch%
     - also used to create the logfile (filename: %os%.log)
%channel% - update channel you are testing (Example: betatest, releasetest, beta, release)
%build% - (optional) only test the given build under the folder specified with %os%

Results

The results for all update tests will be available under the 'logs' folder as separate text files. Open each log and scroll to the bottom to find the results. These can be copied to the results wiki page of the release.