TPS, or Testing and Profiling tool for Sync, is a multi-profile test automation framework for Firefox Sync.
Installation
TPS comes with an installation script that sets up TPS inside a virtualenv:
hg clone https://hg.mozilla.org/mozilla-central cd mozilla-central/testing/tps ./create_venv.py /path/to/create/virtualenv
Once installed, you will need to activate the virtualenv before using TPS. The command for doing this is:
source /path/to/virtualenv/bin/activate
or, on Windows (using Mozilla build tools/msys):
source /path/to/virtualenv/Scripts/activate
You can then launch 'runtps' to run TPS.
To quit the virtualenv, type 'deactivate' at the virtualenv prompt.
Running Tests
TPS can be invoked using the 'runtps' command. Command arguments are:
$ runtps --help Usage: runtps-script.py [options] Options: -h, --help show this help message and exit --binary=BINARY path to the Firefox binary, specified either as a local file or a url; if omitted, the PATH will be searched; --configfile=CONFIGFILE path to the config file to use default: none] --debug run in debug mode --ignore-unused-engines If defined, do not load unused engines in individual tests. Has no effect for pulse monitor. --logfile=LOGFILE path to the log file [default: tps.log] --mobile run with mobile settings --pulsefile=PULSEFILE path to file containing a pulse message in json format that you want to inject into the monitor --resultfile=RESULTFILE path to the result file [default: tps_result.json] --testfile=TESTFILE path to the test file to run [default: ../../services/sync/tests/tps/all_tests.json]
For example, to run the test test_sync.js using a specific tinderbox bulid, you would use this command:
runtps --testfile=/path/to/mozilla-central/services/sync/tests/tps/test_sync.js --binary=https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-31.0a1.en-US.linux-i686.tar.bz2
To run a different test against a local build, you might use this command:
runtps --testfile=/path/to/mozilla-central/services/sync/tests/tps/test_client_wipe.js --binary=~/builds/mozilla-central/firefox
Tests are specified relative to the tps directory. Binaries can either be local files, or http: or ftp: url's, in which case the binary is downloaded and installed prior to running the test.
The config file
Several options are set using a configuration file, which is config.json by default. This is located inside the virtualenv created when you ran INSTALL.sh. On Linux, this path to this file will look something like:
/path/to/virtualenv/lib/python2.6/site-packages/tps-0.2.40-py2.6.egg/tps/config.json
On Windows, it will be something like:
/path/to/virtualenv/Lib/site-packages/tps-0.2.40-py2.6.egg/tps/config.json
Be sure to edit this file, and not config.json.in in the TPS source.
config.json is a JSON file with the following format:
{
"sync_account": {
"username": "[email protected]",
"password": "*******",
"passphrase": "r-jwcbc-zgf42-fjn72-p5vpp-iypmi"
},
"fx_account": {
"username": "my_email",
"password": "*******"
},
"email": {
"username": "[email protected]",
"password": "",
"passednotificationlist": ["[email protected]"],
"notificationlist": ["[email protected]"]
},
"auth_type": "fx_account",
"platform": "win32",
"os": "win7",
"es": "localhost:9200",
"serverURL": null,
"testdir": "/c/mozilla/mozilla-central/src/services/sync/tests/tps/",
"extensiondir": "/c/mozilla/mozilla-central/src/services/sync/tps/extensions/"
}
- sync_account - these settings specify the Sync account that will be used during testing. This account will be wiped during testing, so do not use an account that you use for personal use.
- username, password, passphrase - the details of the sync account to use for testing
- fx_account - these settings specify the Firefox account that will be used during testing. This account will be wiped during testing, so do not use an account that you use for personal use.
- username, password - the details of the Firefox Account to use for testing
- email - these settings specify details of the email account used to send email notifications of test results. These settings are only used when runtps is launched with the --email-results option.
- username/password - account credentials of the mozilla.com email address used to send email notifications
- passednotificationlist - list of email addresses to notify on completion of successful test runs
- notificationlist - list of email addresses to notify on completion of failed test runs
- auth_type - type of authentication, which is by default fx_account. Otherwise also the old sync authentication can be used by specifying 'sync_account'.
- platform - the platform of the machine running the test, should be one of: win32, win64, linux, linux64, macosx, macosx64
- os - free-form OS string
- serverURL - the sync server; leave empty to use the default server
- testdir - absolute path to the TPS tests inside the mozilla-central tree, this value is generated by the INSTALL.sh script
- extensiondir - absolute path to the TPS extensions inside the mozilla-central tree, this value is generated by the INSTALL.sh script
Manifest files
You can run multiple tests in sequence by specifying a test manifest file with the --testfile option. Test manifests are JSON files which contain a list of test files. There is one such file, services/sync/tests/tps/all_tests.json, in the TPS repo:
{ "tests": [
"test_sync.js",
"unittests/test_prefs.js",
"unittests/test_tabs.js",
"unittests/test_passwords.js",
"unittests/test_history.js",
"unittests/test_formdata.js",
"tests/test_bug530717.js",
"tests/test_bug531489.js",
"tests/test_bug538298.js",
"tests/test_bug556509.js",
"tests/test_bug562515.js",
"tests/test_bug563989.js",
"tests/test_bug535326.js",
"tests/test_bug501528.js",
"tests/test_bug575423.js",
"tests/test_bug546807.js",
"tests/test_history_collision.js",
"tests/test_privbrw_formdata.js",
"tests/test_privbrw_passwords.js",
"tests/test_privbrw_tabs.js",
"tests/test_bookmarks_in_same_named_folder.js",
"tests/test_client_wipe.js",
"tests/test_special_tabs.js"
]
}
Continuous integration testing
The continuous integration system for TPS is Coversheet. If you run it without specifying a binary, TPS will start a Mozilla Pulse listener, which waits for messages from Pulse that indicate that a tinderbox build has been made. When it receives such a message, it downloads and installs the build, then initiates two sets of tests using the test_manifest.json file.
- Set 1: Default Sync server, desktop prefs
- Set 2: Default Sync server, mobile prefs
It's also possible artificially inject a "fake" Pulse message into TPS, so that it will run the above set of tests against an arbitrary build. You can specify the path to this fake Pulse message using the --pulsefile option; such files look like this:
{
"buildtype": "opt",
"tree": "mozilla-central",
"buildurl": "https://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1309404226/firefox-7.0a1.en-US.win32.zip"
}
After completing these tests against the build specified in the pulse file, TPS will listen for real Pulse messages.
Results of TPS runs via the CI system are getting send to autolog which can be found at https://brasstacks.mozilla.com/autolog/.