Revision 1006621 of Automated Testing of B2G OS

<p>{{FirefoxOSSidebar}}</p><p>{{B2GMain}}</p>

<div class="summary">
<p>Given that Firefox OS is still under development, and support for new hardware is going to be forthcoming for the foreseeable future, it's important to know how to test it. <span class="seoSummary">This page offers articles that provide information about various aspects of testing Firefox OS, including running different tests, automation, and result reporting and tracking. </span></p>
</div>

<h2 id="Getting_started">Getting started</h2>

<dl>
 <dt><a href="/en-US/Firefox_OS/Running_Tests_on_Firefox_OS_for_Developers">Running tests on Firefox OS: A guide for developers</a></dt>
 <dd>A quick, developer-focused guide to getting started with running the tests. This is where you should start if you are not experienced in running Mozilla's tests and automation systems. If you are, then you will probably have an idea of what tests you want to run and how, and you can skip on to the more specific detailed guides below.</dd>
 <dt><a href="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/MozITP">The Mozilla integrated tools package</a></dt>
 <dd>The Mozilla integrated tools package (MozITP) is a one-stop shop for setting up Firefox OS-related tools, which can handle automatic Testing on Mulet or a real device, flashing TaskCluster images, flashing with the <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Building_and_installing_Firefox_OS/B2G_installer_add-on">B2G installer add-on</a>, and running the TV and phone versions of Mulet.</dd>
</dl>

<h2 id="Gaia_tests">Gaia tests</h2>

<p>These articles cover the primary test suites designed to put Gaia through its paces.</p>

<dl>
 <dt><a href="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/gaia-ui-tests" title="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/gaia-ui-tests">Gaia UI tests</a></dt>
 <dd>Python tests for Gaia UI interactions and features.</dd>
 <dt><a href="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/Gaia_integration_tests">Gaia integration tests</a></dt>
 <dd>JavaScript integration tests for Gaia, based on Marionette.</dd>
 <dt><a href="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/Gaia_unit_tests" title="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/Gaia_unit_tests">Gaia unit tests</a></dt>
 <dd>Gaia unit tests with no UI interaction; written in JavaScript, not Marionette-based.</dd>
 <dt><a href="/en-US/Firefox_OS/Automated_testing/Raptor">Raptor: Performance Tools for Gaia</a></dt>
 <dd>Raptor is a tool for measuring performance specifically on Firefox OS, which aims to improve upon the existing performance testing tools.</dd>
 <dt><a href="https://wiki.mozilla.org/Project_Eideticker" title="https://github.com/mozilla/eideticker">Eideticker</a></dt>
 <dd>Provides performance measurements for Firefox OS based on video captures.</dd>
 <dt><a href="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/MTBF_tests">MTBF test</a></dt>
 <dd>Mean Time Between Failure. This is a test suite that runs on device for long duration, attempting to find problems with Gaia uptime and stability. (Currently, it is owned by Taiwan QA team and still a developing test framework)</dd>
</dl>

<h2 id="B2G_tests">B2G tests</h2>

<p>The guides below cover a number of different test harnesses that test various aspects of B2G functionality.</p>

<dl>
 <dt><a href="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/Mochitests" title="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/Mochitests">Mochitests</a></dt>
 <dd>Gecko functional and API tests; HTML &amp; JS based.&nbsp; No Gaia interaction.</dd>
 <dt><a href="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/Reftests" title="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/Reftests">Reftests</a></dt>
 <dd>Gecko rendering correctness tests.</dd>
 <dt><a href="/en-US/docs/Marionette/Marionette_JavaScript_Tests" title="/en-US/docs/Marionette/Marionette_JavaScript_Tests">WebAPI tests</a></dt>
 <dd>Gecko JS-based WebAPI tests; most of these require an emulator.</dd>
 <dt><a href="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/XPCShell" title="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/XPCShell">xpcshell tests</a></dt>
 <dd>Headless tests of Gecko XPCOM APIs.</dd>
 <dt><a href="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/Cppunit_Tests" title="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/Cppunit_Tests">cppunit tests</a></dt>
 <dd>Headless C++ unit tests.</dd>
</dl>

<h2 id="Supporting_documentation">Supporting documentation</h2>

<p>This section provides links to some of the supporting technologies that underpin Mozilla's tests, which you may want to find more information about.</p>

<dl>
 <dt><a href="/en-US/docs/Marionette" title="/en-US/docs/Marionette">Marionette</a></dt>
 <dd>A remote test driver based on Selenium WebDriver.</dd>
 <dt><a href="/en-US/docs/Marionette/Marionette_JavaScript_Tools" title="/en-US/docs/Marionette/Marionette_JavaScript_Tools">Marionette JavaScript tools</a></dt>
 <dd>A node.js-based runner for Marionette.</dd>
 <dt><a href="/en-US/docs/Marionette/Python_Marionette" title="/en-US/docs/Marionette/Python_Marionette">Marionette Python client</a></dt>
 <dd>A Python runner for Marionette.</dd>
 <dt><a href="https://wiki.mozilla.org/Build:TryServer">Try server</a></dt>
 <dd>Mozilla's standard server for testing patches before checking them into the core repository. See also the <a href="https://trychooser.pub.build.mozilla.org/">TryChooser Syntax Builder</a>.</dd>
</dl>

<div class="note">
<p><strong>Note</strong>: If you want to run Marionette against a production build (to run gaia integration tests, gaia-ui-tests, etc.), you can <a href="https://github.com/mozilla-b2g/marionette-extension">install Marionette as an extension</a> (this currently only works for 1.3 builds, but more support will be added soon.)</p>
</div>

<h2 id="Continuous_integration_and_result_reporting">Continuous integration and result reporting</h2>

<p>The following articles cover the continuous integration and result reporting mechanisms Mozilla uses to store and intepret test data.</p>

<dl>
 <dt><a href="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/Treeherder">Treeherder</a></dt>
 <dd>Understand the tests and builds that run on Treeherder.</dd>
 <dt><a href="https://raptor.mozilla.org/">Raptor</a></dt>
 <dd>Visualization of performance tests run against the <a href="/en-US/Firefox_OS/Automated_testing/Raptor">Raptor</a> tool.</dd>
 <dt><a href="/en-US/docs/Mozilla/Firefox_OS/Automated_testing/Test_Execution_Chart" title="/en-US/docs/Mozilla/Firefox_OS/Testing/Test_Execution_Chart">Test execution chart</a></dt>
 <dd>A chart showing which tests are being run — on what devices and where — and which platforms are supported for each test.</dd>
</dl>