Please note, this is a STATIC archive of website developer.mozilla.org from 03 Nov 2016, cach3.com does not collect or store any user information, there is no "phishing" involved.

TPS, or Testing and Profiling tool for Sync, is a multi-profile test automation framework for Firefox Sync.

Installation

TPS comes with an installation script that sets up TPS inside a virtualenv:

  hg clone https://hg.mozilla.org/mozilla-central
  cd mozilla-central/testing/tps
  ./create_venv.py /path/to/create/virtualenv

Once installed, you will need to activate the virtualenv before using TPS. The command for doing this is:

  source /path/to/virtualenv/bin/activate

or, on Windows (using Mozilla build tools/msys):

  source /path/to/virtualenv/Scripts/activate

You can then launch 'runtps' to run TPS.

To quit the virtualenv, type 'deactivate' at the virtualenv prompt.

Running Tests

TPS can be invoked using the 'runtps' command.  Command arguments are:

$ runtps --help
Usage: runtps-script.py [options]

Options:
  -h, --help            show this help message and exit
  --binary=BINARY       path to the Firefox binary, specified either as a
                        local file or a url; if omitted, the PATH will be
                        searched;
  --configfile=CONFIGFILE
                        path to the config file to use default: none]
  --debug               run in debug mode
  --ignore-unused-engines
                        If defined, do not load unused engines in individual
                        tests. Has no effect for pulse monitor.
  --logfile=LOGFILE     path to the log file [default: tps.log]
  --mobile              run with mobile settings
  --pulsefile=PULSEFILE
                        path to file containing a pulse message in json format
                        that you want to inject into the monitor
  --resultfile=RESULTFILE
                        path to the result file [default: tps_result.json]
  --testfile=TESTFILE   path to the test file to run [default:
                        ../../services/sync/tests/tps/all_tests.json]

For example, to run the test test_sync.js using a specific tinderbox bulid, you would use this command:

runtps --testfile=/path/to/mozilla-central/services/sync/tests/tps/test_sync.js --binary=https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-31.0a1.en-US.linux-i686.tar.bz2

To run a different test against a local build, you might use this command:

runtps --testfile=/path/to/mozilla-central/services/sync/tests/tps/test_client_wipe.js --binary=~/builds/mozilla-central/firefox

Tests are specified relative to the tps directory.  Binaries can either be local files, or http: or ftp: url's, in which case the binary is downloaded and installed prior to running the test.

The config file

Several options are set using a configuration file, which is config.json by default.  This is located inside the virtualenv created when you ran INSTALL.sh.  On Linux, this path to this file will look something like:

/path/to/virtualenv/lib/python2.6/site-packages/tps-0.2.40-py2.6.egg/tps/config.json

On Windows, it will be something like:

/path/to/virtualenv/Lib/site-packages/tps-0.2.40-py2.6.egg/tps/config.json

Be sure to edit this file, and not config.json.in in the TPS source.

config.json is a JSON file with the following format:

{
  "sync_account": {
    "username": "[email protected]",
    "password": "*******",
    "passphrase": "r-jwcbc-zgf42-fjn72-p5vpp-iypmi"
  },
  "fx_account": {
    "username": "my_email",
    "password": "*******"
  },
  "email": {
    "username": "[email protected]",
    "password": "",
    "passednotificationlist": ["[email protected]"],
    "notificationlist": ["[email protected]"]
  },
  "auth_type": "fx_account",
  "platform": "win32",
  "os": "win7",
  "es": "localhost:9200",
  "serverURL": null,
  "testdir": "/c/mozilla/mozilla-central/src/services/sync/tests/tps/",
  "extensiondir": "/c/mozilla/mozilla-central/src/services/sync/tps/extensions/"
 }
  • sync_account - these settings specify the Sync account that will be used during testing.  This account will be wiped during testing, so do not use an account that you use for personal use.
    • username, password, passphrase - the details of the sync account to use for testing
  • fx_account - these settings specify the Firefox account that will be used during testing.  This account will be wiped during testing, so do not use an account that you use for personal use.
    • username, password - the details of the Firefox Account to use for testing
  • email - these settings specify details of the email account used to send email notifications of test results.  These settings are only used when runtps is launched with the --email-results option.
    • username/password - account credentials of the mozilla.com email address used to send email notifications
    • passednotificationlist - list of email addresses to notify on completion of successful test runs
    • notificationlist - list of email addresses to notify on completion of failed test runs
  • auth_type - type of authentication, which is by default fx_account. Otherwise also the old sync authentication can be used by specifying 'sync_account'.
  • platform - the platform of the machine running the test, should be one of: win32, win64, linux, linux64, macosx, macosx64
  • os - free-form OS string
  • serverURL - the sync server; leave empty to use the default server
  • testdir - absolute path to the TPS tests inside the mozilla-central tree, this value is generated by the INSTALL.sh script
  • extensiondir - absolute path to the TPS extensions inside the mozilla-central tree, this value is generated by the INSTALL.sh script

Manifest files

You can run multiple tests in sequence by specifying a test manifest file with the --testfile option.  Test manifests are JSON files which contain a list of test files.  There is one such file, services/sync/tests/tps/all_tests.json, in the TPS repo:

{ "tests": [
    "test_sync.js",
    "unittests/test_prefs.js",
    "unittests/test_tabs.js",
    "unittests/test_passwords.js",
    "unittests/test_history.js",
    "unittests/test_formdata.js",
    "tests/test_bug530717.js",
    "tests/test_bug531489.js",
    "tests/test_bug538298.js",
    "tests/test_bug556509.js",
    "tests/test_bug562515.js",
    "tests/test_bug563989.js",
    "tests/test_bug535326.js",
    "tests/test_bug501528.js",
    "tests/test_bug575423.js",
    "tests/test_bug546807.js",
    "tests/test_history_collision.js",
    "tests/test_privbrw_formdata.js",
    "tests/test_privbrw_passwords.js",
    "tests/test_privbrw_tabs.js",
    "tests/test_bookmarks_in_same_named_folder.js",
    "tests/test_client_wipe.js",
    "tests/test_special_tabs.js"
  ]
}

 

Continuous integration testing

The continuous integration system for TPS is Coversheet. If you run it without specifying a binary, TPS will start a Mozilla Pulse listener, which waits for messages from Pulse that indicate that a tinderbox build has been made.  When it receives such a message, it downloads and installs the build, then initiates two sets of tests using the test_manifest.json file.

  • Set 1:  Default Sync server, desktop prefs
  • Set 2:  Default Sync server, mobile prefs

It's also possible artificially inject a "fake" Pulse message into TPS, so that it will run the above set of tests against an arbitrary build.  You can specify the path to this fake Pulse message using the --pulsefile option; such files look like this:

{
  "buildtype": "opt",
  "tree": "mozilla-central",
  "buildurl": "https://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1309404226/firefox-7.0a1.en-US.win32.zip"
}

After completing these tests against the build specified in the pulse file, TPS will listen for real Pulse messages.

Results of TPS runs via the CI system are getting send to autolog which can be found at https://brasstacks.mozilla.com/autolog/.

Writing TPS Tests

TPS Tests

Document Tags and Contributors

Tags: 
 Contributors to this page: teoli, Whimboo, jrgm, [email protected], jgriffin
 Last updated by: Whimboo,