mach

Mach (German for do) is a command-line interface to help developers perform common tasks. The purpose of mach is to help make the developer experience better by making common tasks easier to discover and perform.

Requirements

Mach requires a current version of mozilla-central (or a tree derived from) (mach was committed on 2012-09-26). Mach also requires Python 2.7. mach itself is Python 3 compliant. However, modules used by mach likely aren't Python 3 compliant just yet. Stick to Python 2.7.

Running

From the root of the source tree checkout, you should just be able to type:

$ ./mach

If all is well, you should see a help message.

For full help:

$ ./mach help

Try building the tree:

$ ./mach build

If you get error messages, make sure that you have all of the build requisites for your system.

If it works, you can look at compiler warnings:

$ ./mach warnings-list

Try running some tests:

$ ./mach xpcshell-test services/common/tests/unit/

Or run an individual test:

$ ./mach mochitest browser/base/content/test/general/browser_pinnedTabs.js

You run mach from the source directory, so you should be able to use your shell's tab completion to tab-complete paths to tests. Mach figures out how to execute the tests for you!

mach and mozconfigs

It's possible to use mach with multiple mozconfig files. mach's logic for determining which mozconfig to use is effectively the following:

1. If a .mozconfig file exists in the current directory, use that.
2. If the MOZCONFIG environment variable is set, use the file pointed to in that variable.
3. If the current working directory mach is invoked with is inside an object directory, the mozconfig used when creating that object directory is used.
4. The default mozconfig search logic is applied.

Here are some examples:

# Use an explicit mozconfig file.
$ MOZCONFIG=/path/to/mozconfig ./mach build

# Alternatively (for persistent mozconfig usage):
$ export MOZCONFIG=/path/to/mozconfig
$ ./mach build

# Let's pretend the MOZCONFIG environment variable isn't set. This will use
# the mozconfig from the object directory.
$ cd objdir-firefox
$ mach build

Adding mach to your shell's search path

If you add mach to your path (by modifying the PATH environment variable to include your source directory, or by copying mach to a directory in the default path like /usr/local/bin) then you can type mach anywhere in your source directory or your objdir.  Mach expands relative paths starting from the current working directory, so you can run commands like mach build . to rebuild just the files in the current directory.  For example:

$ cd browser/devtools
$ mach build webconsole # Rebuild only the files in the browser/devtools/webconsole directory
$ mach mochitest-browser webconsole/test # Run browser-chrome tests from browser/devtools/webconsole/test

Enable tab completion

To enable tab completion in bash, run the following command.  You can add the command to your .profile so it will run automatically when you start the shell:

source /path/to/mozilla-central/python/mach/bash-completion.sh

This will enable tab completion of mach command names, and in the future it may complete flags and other arguments too.  Note: Mach tab completion will not work when running mach in a source directory older than Firefox 24.

For zsh, you can call the built-in bashcompinit function before sourcing:

autoload bashcompinit
bashcompinit
source /path/to/mozilla-central/python/mach/bash-completion.sh

Frequently Asked Questions

Why should I use mach?

You should use mach because it provides a better and more unified developer experience for working on Mozilla projects. If you don't use mach, you have to find another solution for the following problems:

• Discovering what commands or make targets are available (mach exposes everything through mach help)
• Making more sense out of command output (mach offers terminal colorization and structured logging)
• Getting productive tools in the hands of others (mach advertises tools to people through mach help - people don't need to discover your tool from a blog post, wiki page, or word of mouth)

Are there any known issues?

Several. Mach is still relatively young and there are a number of bugs and numerous areas for improvement. Some larger known issues include:

• MinTTY (alternative terminal emulator on Windows) doesn't work
• Text encoding issues (especially on Windows where Latin-1 is not the default system encoding).
• Failed commands spew lots of extra error output (e.g. you will see a big mach error message when all that happened was an invoked command returned a non-0 exit code (possibly expectedly).

Generally, mach is known to work pretty well without issues for most people.

How do I report bugs?

Bugs against the mach core can be filed at https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=mach.

How is building with mach different from building with client.mk, from using make directly?

Currently, mach build simply invokes client.mk. There are no differences in terms of how the build is performed (well, at least there shouldn't be).

Mach does offer some additional features over manual invocation of client.mk:

• If on Windows, mach will automatically use pymake instead of GNU make, as that is preferred on Windows.
• mach will print timings with each line of output from the build. This gives you an idea of how long things take.
• mach will colorize terminal output (on terminals that support it - typically most terminals except on Windows)
• mach will scan build output for compiler warnings and will automatically record them to a database which can be queried with mach warnings-list and mach warnings-summary. Not all warnings are currently detected. Do not rely on mach as a substitute for raw build output.
• mach will invoke make in silent mode. This suppresses excessive (often unncessary) output.

Is mach a build system?

No. Mach is just a generic command dispatching tool that happens to have a few commands that interact with the real build system. Historically, mach was born to become a better interface to the build system. However, its potential beyond just build system interaction was quickly realized and mach grew to fit those needs.

Does mach work with mozconfigs?

Yes! You use mozconfigs like you have always used them.

Does mach have its own configuration file?

Not yet. It will likely have one some day.

Should I implement X as a mach command?

There are no hard or fast rules. Generally speaking, if you have some piece of functionality or action that is useful to multiple people (especially if it results in productivity wins), then you should consider implementing a mach command for it.

Some other cases where you should consider implementing something as a mach command:

• When your tool is a random script in the tree. Random scripts are hard to find and may not conform to coding conventions or best practices. Mach provides a framework in which your tool can live that will put it in a better position to succeed than if it were on its own.
• When the alternative is a make target. The build team generally does not like one-off make targets that aren't part of building (read: compiling) the tree. This includes things related to testing and packaging. These weigh down make files and add to the burden of maintaining the build system. Instead, you are encouraged to implement ancillary functionality in not make (preferably Python). If you do implement something in Python, hooking it up to mach is often trivial (just a few lines of proxy code).

How does mach fit into the modules system?

Mozilla operates with a modules governance system where there are different components with different owners. There is not currently a mach module. There may or may never be one. Mach is just a generic tool. The mach core is the only thing that could fall under perview of a module and an owner.

Even if a mach module were established, mach command modules (see below) would likely never belong to it. Instead, mach command modules are owned by the team/module that owns the system they interact with. In other words, mach is not a power play to consolidate authority for tooling. Instead, it aims to expose that tooling through a common, shared interface.

Who do I contact for help or to report issues?

The maintainer of mach is Gregory Szorc (gps on IRC or [email protected]). You can also ask questions in #mach, #developers. Or, if you say mach in any IRC channel gps is in, he will probably notice.

Can I use mach outside of mozilla-central?

Yes! The mach core is in mozilla-central inside the python/mach directory and available on PyPI at https://pypi.python.org/pypi/mach/.

Can I use mach with B2G?

Yes again! After you have cloned the B2G repo run config.sh to pull in gecko. The bootstrap script will then scan the gecko repository for mach commands that also apply to the B2G repo. There's also an external module you can install called b2g-commands which provides B2G specific mach commands.

Is there a logo for mach?

Not yet. gps would like the logo to be of a unicorn breaking the sound barrier (mach speed) in front of a rainbow. Contributions are welcome.

mach Architecture

Under the hood mach is a generic command dispatching framework which currently targets command line interfaces (CLIs). You essentially have a bunch of Python functions saying "I provide command X" and mach hooks up command line argument parsing, terminal interaction, and dispatching.

There are 3 main components to mach:

1. The mach core.
2. Mach commands
3. The mach driver

The mach core is the main Python modules that implement the basic functionality of mach. These include command line parsing, a structured logger, dispatching, and utility functions to aid in the implementation of mach commands.

Mach commands are what actually perform work when you run mach. Mach has a few built-in commands. However, most commands aren't part of mach itself. Instead, they are registered with mach.

The mach driver is the mach command line interface. It's a Python script that creates an instance of the mach core, registers commands with it, then tells the mach core to execute.

The canonical source repository for the mach core is the python/mach directory in mozilla-central. The main mach routine lives in main.py. The mach driver is the mach file in the root directory of mozilla-central. As you can see, the mach driver is a shim that calls into the mach core.

As you may have inferred, mach is implemented in Python. Python is our tooling programming language of choice at Mozilla. Mach is also Python 3 compliant (at least it should be).

Adding Features to mach

Most mach features come in the form of new commands. Implementing new commands is as simple as writing a few lines of Python and registering the created file with mach.

The first step to adding a new feature to mach is to file a bug. You have the choice of filing a bug in the Core :: mach component or in any other component. If you file outside of Core :: mach, please add [mach] to the whiteboard.

Mach is relatively new and the API is changing. So, the best way to figure out how to implement a new mach command is probably to look at an existing one.

Start by looking at the source for the mach driver. You will see a list defining paths to Python files (likely named mach_commands.py). These are the Python files that implement mach commands and are loaded by the mach driver. These are relative paths in the source repository. Simply find one you are interested in and dig in!

mach Command Providers

A mach command provider is simply a Python module. When these modules are loaded, mach looks for specific specific signatures to detect mach commands. Currently, this is implemented through Python decorators. Here is a minimal mach command module:

from __future__ import print_function, unicode_literals

from mach.decorators import (
    CommandArgument,
    CommandProvider,
    Command,
)

@CommandProvider
class MachCommands(object):
    @Command('doit', description='Run it!')
    @CommandArgument('--debug', '-d', action='store_true',
        help='Do it in debug mode.')
    def doit(self, debug=False):
        print('I did it!')

From mach.decorators we import some Python decorators which are used to define what Python code corresponds to mach commands.

The decorators are:

@CommandProvider

This is a class decorator that tells mach that this class contains methods that implement mach commands. Without this decorator, mach will not know about any commands defined within, even if they have decorators.

@Command

This is a method decorator that tells mach that this method implements a mach command. The arguments to the decorator are those that can be passed to the argparse.ArgumentParser constructor by way of sub-commands.

@CommandArgument

This is a method decorator that tells mach about an argument to a mach command. The arguments to the decorator are passed to argparse.ArgumentParser.add_argument().

The class and method names can be whatever you want. They are irrelevant to mach.

An instance of the @CommandProvider class is instantiated by the mach driver if a command in it is called for execution. The __init__ method of the class must take either 1 or 2 arguments (including self). If your class inherits from object, no explicit __init__ implementation is required (the default takes 1 argument). If your class's __init__ takes 2 arguments, the second argument will be an instance of mach.base.CommandContext. This object holds state from the mach driver, including the current directory, a handle on the logging manager, the settings object, and information about available mach commands.

The arguments registered with @CommandArgument are passed to your method as keyword arguments using the **kwargs calling convention. So, you should define default values for all of your method's arguments.

The return value from the @Command method should be the integer exit code from the process. If not defined or None, 0 will be used.

Registering mach Command Providers

Once you've written a Python module providing a mach command, you'll need to register it with mach. There are two ways to do this.

If you have a single file, the easiest solution is probably to register it as a one-off inside build/mach_bootstrap.py. There should be a Python list of paths named MACH_MODULES or similar. Just add your file to that list, run mach help and your new command should appear!

Submitting a mach Command for Approval

Once you've authored a mach command, submit the patch for approval. Please flag [email protected] for review.

Mach Command Modules Useful Information

Command modules are not imported into a reliable Python package/module "namespace." Therefore, you can't rely on the module name. All imports must be absolute, not relative.

Because mach command modules are loaded at mach start-up, it is important that they be lean and not have a high import cost. This means that you should avoid global import statements as much as possible. Instead, defer your import until inside the @Command decorated method.

Mach ships with a toolbox of mix-in classes to facilitate common actions. See python/mach/mach/mixin. If you find yourself reinventing the wheel or doing something you feel that many mach commands will want to do, please consider authoring a new mix-in class so your effort can be shared!