Treeherder is where test results are reported for tests being run in buildbot continuous integration. We test builds and run tests against all try and trunk branches, including b2g-inbound, mozilla-central, and Firefox OS release branches. This article explains a bit more about Treeherder, and provides useful terminology.
Note: See the Test Execution Chart to find out which tests are being run where.
Gaia-Try: Git pull requests and Treeherder
Whenever a pull request is created against mozilla-b2g's gaia repository, Mozilla picks up that branch and runs the Gaia suite of tests against it. This is very similar to how Travis works.
The Try Server Hook will comment in your pull request with a link to Treeherder job it has created against your pull request. From here you can review the stability of your patch and whether it is suitable to be merged.
Firefox OS builds for Device and the Desktop Firefox OS simulator
For each commit, Treeherder will attempt to create builds for several different devices, including Hamachi and Flame and also desktop B2G (Firefox OS simulator) builds for linux64 and macos64.
GIP Device tests reporting to Jenkins
For various reasons (notably test runtime) the device tests cannot integrate cleanly with Treeherder. Instead, a standalone Jenkins consumes the Treeherder builds and runs device tests, but with less frequency and on fewer trees. The results and artifacts like output report and logcat are reported back to treeherder but are hidden. Click the "Show hidden jobs" button on your Treeherder job to see the test results.
Running Treeherder Try jobs against Gaia builds
For testing Gecko patches against Firefox OS and Gaia you can push to Try as normal but use the correct trychooser syntax to make sure that Firefox OS and Gaia tests are run. An additional advantage of this is that you can test Gecko and Gaia patches simultaneously by pointing Try to your own Gaia fork. it requires using hg. If you don't want to use hg directly, you can use moz-git-tools.
If you've never pushed to Try, you'll want to read the basic How to push to Try instructions first.
Like all try jobs, this requires Level 1 commit access. If you want to run all tests on B2G Desktop builds with your custom Gaia fork, the correct trychooser syntax is as follows:
try: -b o -p linux64_gecko,linux32_gecko,macosx64_gecko -u all -t none
Treeherder glossary
This section lists some common terms and syntax.
Types of Builds
- "B2G Desktop <platform>" rows: These refer to b2g desktop builds.
- "B2G Device Image" row: These are builds for full-stack b2g images, for both devices (like the Flame) and emulators.
Types of tests
- Gu: These are Gaia unit tests.
- Gij: These are Gaia integration tests written in JavaScript.
- Gip: These are Gaia UI tests written in Python.
- M: These are Mochitests. These have numbers associated with them since they execute tests in chunks.
- R, J and C: These are Reftests, JSreftests, and Crashtests. These have numbers associated with them since they execute tests in chunks.
- X: These are XPCShell tests.
- Mn: These are Marionette unit tests (tests the Marionette framework itself).
- Mnw: These are Marionette unit and WebAPI tests.
Hacking on Treeherder
There are development notes and links to source code at https://wiki.mozilla.org/Auto-tools/Projects/Treeherder.