Gaia UI Tests: Running Tests

Gaia UI テスト は、実際の端末やエミュレーター、B2G Desktop クライアントで実行する性能をもって開発されています。もっとずっと簡単な方法は、B2G デスクトップに対して実行させることです。あなたがお使いのFirefox OS のバージョンと対して実行させる為に、Gaia の正しいブランチを選ばないといけないことを、忘れないで下さい。

B2G デスクトップ上でテスト

B2G クラインアントは、次のようなものです。：デスクトップから楽にFirefox OS上でwebアプリケーションのテスト実行に使える、B2Gのデスクトップ・バージョンです。もし、このテストに詳しくなければ、B2G デスクトップ・クライアント・テストの使用をお読み下さい。この項目では、B2GデスクトップでのUI テストの仕方を教えています。

python setup.py develop

pip install virtualenv
pip install virtualenvwrapper (perhaps need to use sudo here)
export WORKON_HOME=~/Envs
source /usr/local/bin/virtualenvwrapper.sh
echo "source /usr/local/bin/virtualenvwrapper.sh" >> .bash (rc or _profile)

mkvirtualenv gaia-ui-test

Via Marionette, gaiatest is able to launch the B2G Desktop when we start the test. To do this, you need to pass in the path to the B2G binary file and B2G profile. In the below example we have included the $HOME variable as the path to the b2g binary and the b2g profile. This will simply be the location that you unzipped B2G Desktop to after you downloaded it. The command to run all the tests is:

gaiatest --binary=$HOME/b2g/b2g-bin --profile=$HOME/b2g/gaia/profile --testvars=testvars.json --restart --type=b2g ./gaiatest/tests/functional/manifest.ini

Tests that are incompatible with the B2G Desktop client — for example those that make calls and send SMS — are automatically skipped from the test run.

You can run a test directly by calling the file on the command line:

gaiatest --binary=$HOME/b2g/b2g-bin --profile=$HOME/b2g/gaia/profile --testvars=testvars.json --restart ./gaiatest/tests/functional/clock/test_clock_set_alarm.py

Firefox OS デバイスの上でのテスト

実際のFirefox OSデバイス上でのテストは、より複雑です。しかし、テストの中で実際のデバイス上でAPIにアクセスする（+ etc...）ため、より正確な結果を得る事ができます。全てのデバイスAPIや、ハードウェア機能といったものへアクセスできます。しなしながら、Firefox OS の実端末を持っていない場合や、使っている機能が実際のハードウェアを必要としない場合は、エミュレータ内で一般機能をテストすることもできます。

前提条件

テストを実行する

The gaiatest package is needed to run the tests. We need to install this for the branch that we want to run the tests on. Navigate to the gaia/tests/python/gaia-ui-tests/ directory and install gaiatest with this command:

python setup.py develop

Marionette on the device awaits commands on port 2828. We need to forward our local port to the remote port on the device using adb. Run the following command:

adb forward tcp:2828 tcp:2828

Please be reminded that you'll need to start your emulator before the command, if you are not running on a real Firefox OS device.

You can use the following command to execute all the gaia-ui-tests:

gaiatest --address=localhost:2828 --testvars=<TESTVARS_FILE> --restart --type=b2g gaiatest/tests/functional/manifest.ini

Or use the following if you want to run a single test (substituting the last part for the actual test you want to run):

gaiatest --address=localhost:2828 --testvars=<TESTVARS_FILE> --restart gaiatest/tests/functional/clock/test_clock_create_new_alarm.py

After the test, you will see all the pass/fail results and stacktraces for each file.

Gecko とGaia 開発者向けの Treeherder を用いてテストを行う

In this section we will focus upon replicating a Treeherder test locally. Before testing locally with a Treeherder configuration, you need to be aware that it builds a Gaia profile separately. The profile that comes packaged with the Treeherder build or rel-eng build is NOT the same Gaia profile that Treeherder uses.

Treeherder 設定を使って Gaia プロファイルを作成する

Before building a profile we need to check the environment variables that were used to build the profile. Searching in the "brief log" of the Gu test run, find this command block:

Using env: {'DEBUG': '0',
            'DESKTOP': '0',
            'DESKTOP_SHIMS': '0',
            'NOFTU': '0'}

These settings can drastically change the way desktop B2G behaves. It is very important to match Treeherder's settings.

The next step is to checkout the git commit that matches the hg commit that Treeherder ran with. You can also find the hg commit in the "brief log" of the Treeherder run but you will need to correlate that with the git commit yourself.

The gaia test repository includes a make script for building a Gaia profile. Navigate to the Gaia repo that you have cloned locally, and build the profile using the environment variables we found on Treeherder:

DEBUG=0 DESKTOP=0 DESKTOP_SHIMS=0 NOFTU=0 make

テストを実行する

Using the same git commit that Treeherder used to run the test, go to the gaia/tests/python/gaia-ui-tests directory and install gaiatest using the following command:

python setup.py develop

Now that you have built the Gaia profile, the only significant difference to running these tests is that you must direct gaiatest to the binary location and profile location.

Inside the gaia-ui-tests directory, run the below command, with the following substitutions made:

• $BINARY_PATH: The path to the b2g-bin file in the directory you have built or unpacked desktop B2G to.
• $PROFILE_PATH: The path to your locally build Gaia profile. If built from the Gaia repository it will be ~/gaia/profile.

gaiatest --binary $BINARY_PATH/b2g/b2g-bin --profile $PROFILE_PATH/profile --restart --testvars testvars.json gaiatest/tests/manifest.ini --type=b2g

入り組んだ Treeherder ジョブの中から HTML レポートを見つける

All Treeherder jobs (both pass and fail) have an HTML report output generated and stored.

1. For the commit and Gip job, click the chunk of interest (e.g., f1, f2).
2. In the status bar at the bottom of the screen find the section titled: artifact uploaded output.html.
3. Click link to view the report.

既に実行済みのデスクトップ B2G バイナリをテストする

In this section we will look at running the tests against an already running Desktop B2G binary with modified source code.

テストを実行する

Running tests against an existing b2g process is quite easy — you've already gone through the hard parts to build B2G!

If you have not already installed gaiatest then do so now by going to gaia/tests/python/gaia-ui-tests and running the following:

python setup.py develop

Once you have started the b2g binary process and attached your debugger you need to direct gaiatest to the port using the --address command. Do not use --binary or --profile, otherwise gaiatest will just send commands directly to the port without attempting to start or close the binary.

gaiatest --testvars testvars.json gaiatest/tests/functional/test_that_is_being_debugged.py

テスト変数を設定する

You can customize variables used during the test using a testvars.json file. The gaia-ui-tests will wipe the databases on your phone in order to give the test a clean profile to run against. Gaiatest contains protection against running and wiping your device's data. Please read the warning before you set up your test variables file.

We use the --testvars option to pass in local variables, particularly those that cannot be checked into the repository. For example in gaia-ui-tests these variables can be your private login credentials, phone number, or details of your WiFi connection.

To use it, copy testvars_template.json to a different filename but add it into .gitignore so you don't check it into your repository.

When running your tests add the argument: --testvars=(filename).json

(端末用の) testvars.json  を使って WiFi 設定する方法

By setting the WiFi key in testvars.json you can configure the Firefox OS Settings database with your WiFi configuration. Then the test cases can use your WiFi connection.

No WiFi authentication:

"wifi": { "ssid": "MyNetwork"} 

WEP authentication:

"wifi": { "ssid": "MyNetwork", "keyManagement": "WEP", "wep": "MyPassword" } 

WPA-PSK authentication:

"wifi": { "ssid": "MyNetwork", "keyManagement": "WPA-PSK", "psk": "MyPassword" } 

トラブルシューティング

この節では、いくつかのよくあるエラーメッセージを、潜在的な解決策を挙げつつ解説します。

テスト開始前によくある問題