Developing tests

So, you find that you need to develop a test. Perhaps you want to write tests. Perhaps you are not so enthusiastic and, really, you want to get back to writing "real" code. That is ok. You are probably being asked to write tests. But really this is not about you. It is about, or should be about, changes that other people are going to make to your code later. You should want to write a test so that, if anyone tries to change your code in the future and screws it up, they and you and others will know about it as soon as possible.

</soapbox>

There are several test harnesses that you can use to write a test, depending on what you are testing:

• Mochitest - using JavaScript, tests can use windows, documents, chrome, all the browser stuff.
• Mozmill - using JavaScript, tests run in the browser scope and can also handle restarts of the application
• reftest - create two different html files that should look the same in browser. Do they?
• xpcshell tests - using JavaScript, no windows, no chrome, just JavaScript and the lower-level stuff.
• stand-alone - using XPCOM and command-line? These are for you.

Where to go, what to learn

Important Links:

• https://developer.mozilla.org
• https://developer.mozilla.org/en/docs/Category:Automated_testing
• https://bugzilla.mozilla.org/describekeywords.cgi
• https://irc.mozilla.org
• https://reporter.mozilla.org
• https://developer.mozilla.org/en/docs/JavaScript
• https://developer.mozilla.org/en/docs/A_re-introduction_to_JavaScript

Tests can be written from different parts of the Mozilla system. Different kinds of tests need to be developed to test different things. As a developer, you should ask yourself what kinds of things make the most sense to you and where you are most comfortable experimenting. Developing tests may be a way to learn how to develop new functionality in Mozilla, but it is something else also. If you are thinking about testing, you should not think about only the obvious ways to do things. We need to stress the common ways that things are done, but we need to walk the lesser trodden paths as well.

There are different approaches to take in learning about the Mozilla environment. The main "DevMo" site, https://developer.mozilla.org, is your most important resource. Make sure that, as your vocabulary increases, as you learn the terms used to describe parts of the system, you remember to search for terms again. You may have found some resources in the site, but you have not found all of the pages that will help you.

It is important to remember to keep looking around to see what is available. Remember to check the "Recent Changes" page above. It will list the things that are being changed right now, which will tell you what issues people are concerned about and it will point you to people who may be willing to discuss an issue and may want to help you make an impact on an issue that interests you.

Bugs are an important source of information. Some people believe that bugs are the primary documentation of all Mozilla functionality. But it is not always easy to read bugs and make sense of the problem. There are links on the front of the Bugzilla site to "Hot Bugs" or "Bugs Filed Today". Keep in mind that the bugs one sees in these reports may not be the easiest to understand.

The "Describe Keywords" link above may be valuable. Looking at bugs may not help find the "big picture" problems, but the keywords point to some of the cross-cutting issues. Once one finds a bug or bugs that one is interested in, one can look at other bugs in that Component. As you read bugs, some will make more sense than others. Read other bugs originated by the people who write bugs that make sense to you. These people may be able to answer e-mails and may be available on the irc channels on the IRC site listed above.

It may help to change the set of columns in the bug lists. One does this by going to the bottom of a list of bugs and hitting the "Change Columns" link. The default set of columns may not be the most helpful for new users of the system. Try including 'Component' and 'Changed', which will help show which bugs are recently edited, to see the list in a different way. The default list of columns works for some people and not for others.

There are a few ways to find HTML and web pages that cause problems. The reports in the "Reporter" site list web pages for which people have reported problems. Also, one can search for bugs that have html in their attachments. It may not be very obvious how one does this. Go to Bugzilla, hit the Search link and switch to the "Advanced Search" pane. All the way at the bottom of the page is section titled "Advanced Searching Using Boolean Charts". Either select "Attachment is a URL" from the first pull-down menu, or select from all three to say "Attachment mime type" "is equal to" "text/html".

Reftest

If you are a visual person, a visual thinker, or if your primary strength is using HTML and CSS, you may want to focus on developing tests with reftest. In reftest, one can test a fragment of HTML/CSS by putting it in a "test" page, and then creating other HTML/CSS that is, presumably, simpler and produces the same visual effect. This should go into a "reference" page, and the reftest harness verifies that these two pages render the same. This approach may seem odd at first, but if you think about HTML, there is always more than one way to achieve a given visual effect. This is even more true with HTML and CSS.

Just to give an extremely simple example, what are the different kinds of HTML that will produce a single space in a web page? Looking at the HTML 4.01 spec, one can see 4 different space characters and one can see that any arbitrary series of these four characters collapses to one visible space character. There are many test pages that can be written to check only against a reference page that contains nothing but a character, a space, and then another character.

An extremely simple reftest would consist of two html pages.

A line would then be included in one of the existing "reftest.list" files that would say:

== space-collapses-test.html space-collapses-ref.html

When the reftests are then run, the two files would be rendered and it would be confirmed that two spaces do indeed collapse to the same visual display as one non-breaking space.

See the 'Automated Testing' category link in devmo above to find other documents that describe reftest.

Mochitest and xpcshell tests

If you know JavaScript and are comfortable with DOM manipulation, then writing tests for mochitest should be a natural fit. Many, many things that would seem to require user interaction can actually be simulated with JavaScript. Certainly this is true of almost any user interaction that occurs inside the content window of the web browser.

If you want to use JavaScript and you are able to do things without having an actual window, you can use xpcshell. Much of the lower-level functionality available in Mozilla is directly testable with xpcshell. There are many things that could be done is either mochitest or xpcshell. If you are wondering which you should use, keep in mind that you will do best in the environment where you are most comfortable. That being said, though, it is better if a test uses only what it is supposed to test. If a test is targeted to a low-level class and does not require a window, then using mochitest, which provides a window, will cause unnecessary risk. If the window code breaks, should the low-level test fail? Probably not. It is better if only the only tests which fail point to a fault, and if a fault in the system only causes failures in tests targeted to that functionality.

If you are already comfortable with working in the lower layers of the system and are going to work in xpcshell, you may want to skip ahead to some of the documentation referenced above.

The MochiTest harness is very powerful and the flexibility of the JavaScript language and the Mozilla environment combine to suggest that much of the testing that needs to be done now can be done with Mochitest. One will, after all, learn JavaScript in order to do very much with Mozilla applications, and there may be no better place to learn JavaScript than from some of the people that developed the language. In particular, the "Re-Introduction" document above is a gem.

One can write something as a reftest:

color-display-test:html:

<!DOCTYPE html>
<html><head><title>Color Check</title></head>
<body>
<p id="gray">I am some color. Which? I do not know!</p>
<script>
    var p = document.getElementById("gray");
    p.style.color="gray";
</script>
</body>

color-display-ref.html

<!DOCTYPE html>
<html><head><title>Color Check</title></head>
<body>
<p id="gray">I am some color. Which? I do not know!</p>
<script>
    var p = document.getElementById("gray");
    p.style.color="#808080";
</script>
</body>

But the fact that it was possible to write this with a script and not just with a static style element should give a hint to the lesson here. It is also possible to do this much more efficiently with MochiTest. Indeed, I was able to test all the colors specified in the SVG specification, checking that the color names metched with the expected hex values and rgb values also, all in just one script:

<!DOCTYPE HTML>
<html>
<head>
  <title>Test</title>
  <script type="text/javascript" src="/MochiKit/MochiKit.js"></script>
  <script type="text/javascript" src="/tests/SimpleTest/SimpleTest.js"></script>        
  <link rel="stylesheet" type="text/css" href="/tests/SimpleTest/test.css" />
</head>
<body>
<table border="1"><tr><td>colors</td><td id="1">AAAAAAAAAAAA</td></tr></table>
<script>

    function testColors() { }

    function checkColorRender(cname, hex, rgb) {
        var td = document.getElementById("1");

        td.style.color=cname;
        var style = document.defaultView.getComputedStyle(td,"");
        is(style.getPropertyValue('color'), rgb, "\""+cname+"\" should generate color equal to \""+rgb+"\"");

        td.style.color=hex;
        var style = document.defaultView.getComputedStyle(td,"");
        is(style.getPropertyValue('color'), rgb, "\""+hex+"\" should generate color equal to \""+rgb+"\"");
    }
    testColors.check = checkColorRender;

    testColors.check("maroon", "#800000", "rgb(128, 0, 0)");
    testColors.check("red", "#ff0000", "rgb(255, 0, 0)");
    testColors.check("orange", "#ffa500", "rgb(255, 165, 0)");
    testColors.check("yellow", "#ffff00", "rgb(255, 255, 0)");
    testColors.check("olive", "#808000", "rgb(128, 128, 0)");
    <!-- etc, etc, etc... -->
</script></body></html>

It is important to note that MochiTest cannot be used for everything. The test above is checking that the logical values of the colors assigned match what they should. This is not completely guaranteeing that the colors display correctly. But a test targeted at just the color and that the color displays correctly would be very small and there would only need to be one test that verified that. Without a bit of discipline, many tests could waste time re-checking this fact.

JavaScript test-only modules

Reusable JavaScript in the tree is typically organized as JavaScript code modules. It is possible to create modules that are only part of the test suite. This allows for test-specific code to be reused between components cleanly and easily. Previously, one would typically load/include another file into the current executing scope. This old behavior is still supported. However, usage of test-only modules is preferred.

Test-only JS modules are declared in the Makefile by defining the TESTING_JS_MODULES and optionally the TESTING_JS_MODULES_DIR variable. Here is an example:

DEPTH     = ../..
topsrcdir = @top_srcdir@
srcdir    = @srcdir@

include $(DEPTH)/config/autoconf.mk

TESTING_JS_MODULES = mockserver.js common.js
TESTING_JS_MODULES_DIR = foobar

This Makefile will result in the files mockserver.js and common.js from the directory containing the Makefile to be copied into the foobar directory of the test-only modules destination. If TESTING_JS_MODULES_DIR is not defined, files will be installed to the root directory.

Once you have installed test-only modules, they can be imported via the resource://testing-common/ URL prefix. For example:

Components.utils.import("resource://testing-common/foobar/mockserver.js");

Still To Come....

• XULRunner application testing
• Extension testing

Conclusion

If you want to learn how to develop in the Mozilla environment and are developing tests as a means to that end, that is a good thing. It is a good way to learn. But you should be aware that test code is sometimes not the easiest code to figure out. It often does things in an unusual way. This is sometimes the point.

So, look at the code of the other tests. Go to https://lxr.mozilla.org/mozilla and search for the word "test". Look at the code you are trying to test and look for other code, findable around the Internet, that does things you want to try to do. Go to lxr.mozilla.org and browse. Click on the links that take you from one source to an identifier in another source, and another, and another. Be patient.

Remember that things may look odd, or just wrong, but this system works and things are usually the way they are for some reason. The reason may not apply any longer, but you will not know that at first. There is a great deal of stored "tribal" knowledge in the Mozilla community. There are a lot of things that people with experience just know, and they may not realize that these things need to be explained. Go to https://developer.mozilla.org and read the documentation. If you find something that does not make sense, please edit the page until it does.