Using Icecream

Icecream (IceCC) is a distributed compiler with a scheduler that can significantly speed up your macOS or Linux builds. It uses idle compute cycles on nearby machines to parallelise builds. You will see significant improvements with just one remote machine, but the more machines the better.

DistCC vs. IceCC

Both DistCC and IceCC are distributed compilers. They require similar tool chain on all target machines. However, IceCC has a built-in mechanism for distributing a custom toolchain on demand (in a chroot jail). This means that unlike DistCC which requires carefull management and setup of a toolchain, with IceCC each client can use their own toolchain without any setup.

Setup

IceCC on Linux

The following instructions are for setting up IceCC on Linux:

Debian or Ubuntu

sudo apt install icecc

This will automatically start an IceCC Daemon, which should be able to automatically locate and connect to the scheduler running on your network using UDP multi-cast.

Fedora

sudo dnf install icecream
sudo systemctl enable iceccd
sudo systemctl start iceccd

IceCC Client on Mac

The following instructions are for setting up IceCC as a client on macOS. Your macOS machine will not service requests, but will distribute jobs to any x86-64 machines on the network (linux machines), and use them to compile, in addition to local compiles.

This will involve using a custom toolchain which supports cross compilation. This toolchain has been bundled for ease of use.

Install Prerequisites

xcode-select --install
brew install lzo

Download and Install Patched Icecream Daemon and Cross-Compilation Toolchain

git clone https://github.com/mystor/icecc-osx-moztor ~/icecream

Unfortunately, from my testing, the icecream daemon on macOS has trouble discovering schedulers with UDP broadcast, so we explicitly specify the scheduler to connect to when installing the daemon. Install the daemon as follows. If you don't know your scheduler IP, you can check the table at the bottom of this document.

sudo ~/icecream/install.sh SCHEDULER_IP_ADDRESS

Update Mozconfig

# Add these lines to your mozconfig to point it at the compiler wrappers
CC="$HOME/icecream/cc"
CXX="$HOME/icecream/c++"

Using ccache

You can use ccache together with icecc.

On Fedora, no additional configuration should be necessary (since ccache is set up to work transparently out of the box).

On Debian or Ubuntu, because your mozconfig file gets sourced by configure and client.mk, you need to save the variable during configure to expose it to the make environment:

mk_add_options 'export CCACHE_PREFIX=icecc'

export CC=clang
export CXX=clang++

ac_add_options --with-ccache

Note that when using ccache (which you should!) you don’t want to include icecc in the CC or CXX output variables. As we want ccache to be the primary entry-point, we set a prefix to the command line that ccache uses when invoking the compiler.

Building

You will want to run future builds with enough jobs to saturate your cluster. For example:

./mach build -j 100

You can also specify this flag in your mozconfig, to avoid typing it after every `build` command:

mk_add_options MOZ_MAKE_FLAGS="-j100"

You can also use a script like https://gist.github.com/mystor/27beaf86a4868a357dcd58f585b42c6a to get the number of jobs in your network, and set the MOZ_MAKE_FLAGS that way.

Running a scheduler

One machine per network will need to run the IceCream scheduler (icecc-scheduler) program. These computers will be responsible for scheduling jobs in the network and dispatching them to computers to be executed. The following are the IP addresses of the schedulers in each office:

The Icecream monitor

There is a monitor program, icemon which can be run to give a visual interface to the activity currently in the IceCream network. It has some useful, and nice to look at, views, such as the Gnatt view, which provides a satisfying display of the current jobs running on the network.

On some platforms, such as macOS, icemon can't discover the scheduler automatically. If this is the case, use the environment variable USE_SCHEDULER=SCHEDULER_IP.

Frequent problems & debugging

It’s not working, how do I debug this?

Start all processes with -vvv to see tracing information.

My machine shows up in icemon but does not accept job

Your daemon version might be too old, or the client might not have a suitable toolchain to use that node.

Everything is really slow!

You probably either don't have your icecream daemon running (check if a process named iceccd is running), aren't connected to the scheduler, or are not on a LAN wired connection. A good connection to the network is essential for icecream performance.

All out-going jobs failed with exception 23

This means that your machine makes wrong toolchain. Remove a few Ubuntu packages will fix this issue.

sudo apt purge hardening-wrapper hardening-includes

I get build failures due to -Werror when building remotely but not when building locally

Icecream expands all source with the preprocessor before sending it over the wire, so any warnings which are silenced when they are produced by code within macros won't be silenced when building on the remote machine. Try building without --enable-warnings-as-errors to get the build to succeed over icecream.

Alternatively, you could keep --enable-warnings-as-errors, and just disable certain problematic warnings. I found disabling -Wunreachable-code, -Wparentheses-equality, -Wundefined-bool-conversion, -Wunused-value, -Wconstant-logical-operand, -Wtautological-compare, and -Wformat-extra-args to be sufficient.

Related notes

Scratch notes of how the toolchain was generated. These are a bit out date against what we exactly host (builts against trunk and not 3.8). But it should be easy to get running from these instructions:

https://bug1279288.bmoattachments.org/attachment.cgi?id=8761737