Running Thunderbird MozMill tests from packaged tests

If you want to run the MozMill tests that the Thunderbird builders run in automated fashion against a pre-compiled Thunderbird, it is possible to do this.  

This description works for Thunderbird 9.0a1 or newer. You do not need to install mozmill as a separate package anymore, because it is now included in the source (see bug 656736).

Getting Thunderbird and the tests

It is best to download the Thunderbird package and associated tests together. Then you can be sure that the versions of tests correctly match the Thunderbird build you're running against. The lastest release (candidates) builds including the tests can be found there.  There are also folders with the latest beta, earlybird and daily builds. Depending on which version you want, you might have to look in a subdirectory for the correct files.

The two files you want to get are (for a release (candidate) build): 

1. thunderbird-<version>.<package extension>
   • e.g. Linux: thunderbird-9.0.1.tar.bz2
   • Windows: thunderbird-9.0.1.zip (note: its best to use the zip rather than the installer. It can be found in the unsigned/win32/en-US subdirectory, not in the win32/en-US subdirectory)
   • Mac: thunderbird-9.0.1.dmg
2. thunderbird-9.0.1.tests.zip

You can also get this from:

• Tinderbox builds
  • These are short-lived builds produced by the Thunderbird builders, best for if you're trying to cut a regression range down to less than a day.
• Current try-server builds
  • Only really useful if you're testing a specific change that has been pushed to the try server.

Setting up to run the tests

So you've got your two files, now what? First create a sub-directory to work in:

mkdir mozmilltest
cd mozmilltest

Next unpack the tests. This is slightly different per platform. The documentation below uses the example file names from above, just replace them with what you have to run.

Windows

unzip ../thunderbird-9.0.1.zip
unzip ../thunderbird-9.0.1.tests.zip

Mac

wget https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/installdmg.sh
bash installdmg.sh ../thunderbird-9.0.1.dmg
unzip ../thunderbird-9.0.1.tests.zip

Linux

tar -vxjf ../thunderbird-9.0.1.tar.bz2
unzip ../thunderbird-9.0.1.tests.zip

Setup mozmill

You have to create a virtual environment containing the mozmill version included in the tests:

cd mozmill
python resources/installmozmill.py ../mozmill-virtualenv

Running all the tests

Now you're all set up for running the tests, here's how to do it.

Windows

../mozmill-virtualenv/Scripts/python runtestlist.py --binary=../thunderbird/thunderbird.exe --list=mozmilltests.list

Mac

../mozmill-virtualenv/Scripts/python runtestlist.py --binary=../Shredder.app --list=mozmilltests.list 

Note: On Mac, the binary name may be different depending on what you are running, e.g. it could be Thunderbird.app, Miramar.app or something else.

Linux

python runtestlist.py --binary=../thunderbird/thunderbird --list=mozmilltests.list

Running a single test

If you want to run a single test (or all the tests in one folder), then it requires a slightly different command. The examples below show the running of the content-tabs/test-content-tab.js test, but if you wanted to run all the content-tab tests, just replace the last bit with content-tabs.

Windows

../mozmill-virtualenv/Scripts/python runtest.py --binary=../thunderbird/thunderbird.exe -t content-tabs/test-content-tab.js

Mac

../mozmill-virtualenv/Scripts/python runtest.py --binary=../Shredder.app -t content-tabs/test-content-tab.js

Note: On Mac, the binary name may be different depending on what you are running, e.g. it could be Thunderbird.app, Miramar.app or something else.

Linux

python runtest.py --binary=../thunderbird/thunderbird -t content-tabs/test-content-tab.js