Thread Sanitizer

What is Thread Sanitizer?

Thread Sanitizer (TSan) is a fast data race detector for C/C++ programs. It uses a compile-time instrumentation to check all non-race-free memory access at runtime. Unlike other tools, it understands compiler-builtin atomics and synchronization and therefore provides very accurate results with no real false positives. More information on how TSan works can be found on the Thread Sanitizer wiki.

Public Builds

Manual Build

Build prerequisites

LLVM/Clang

The TSan instrumentation is implemented as an LLVM pass and integrated into Clang. The version of Clang you need to use depends on what Firefox release you are compiling

• If you are building from mozilla-central, you must use Clang 3.6 or later.  Clang 3.4 + TSan cannot compile Firefox due to an internal compiler error; some versions of Clang 3.5 packaged by Linux distros do not work for the same reason.  You can check which version of clang you have by running the command:

  clang -v
  

• If you are not building from mozilla-central and are building Firefox 39 or earlier, you must use Clang 3.3.  You can find precompiled binaries for LLVM/Clang 3.3 on the LLVM releases page.

We recommend using Clang 3.6 or later.

Building Firefox

Getting the source

If you don't have a source code repository clone yet, you need to get yourself a clone of mozilla-central.

Adjusting the build configuration

Create the build configuration file .mozconfig with the following content in your mozilla-central directory:

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-ff-tsan
mk_add_options MOZ_MAKE_FLAGS=-j12

# Enable LLVM specific code and build workarounds
ac_add_options --enable-thread-sanitizer
# If clang is already in your $PATH, then these can simply be:
#   export CC=clang
#   export CXX=clang++
export CC="/path/to/clang"
export CXX="/path/to/clang++"

# llvm-symbolizer displays much more complete backtraces when data races are detected.
# If it's not already in your $PATH, then uncomment this next line:
#export LLVM_SYMBOLIZER="/path/to/llvm-symbolizer"

# Add TSan to our compiler flags
export CFLAGS="-fsanitize=thread -fPIC -pie"
export CXXFLAGS="-fsanitize=thread -fPIC -pie"

# Additionally, we need the TSan flag during linking. Normally, our C/CXXFLAGS would
# be used during linking as well but there is at least one place in our build where
# our CFLAGS are not added during linking.
# Note: The use of this flag causes Clang to automatically link the TSan runtime :)
export LDFLAGS="-fsanitize=thread -fPIC -pie"

# These three are required by TSan
ac_add_options --disable-jemalloc
ac_add_options --disable-crashreporter
ac_add_options --disable-elf-hack

# Keep symbols to symbolize TSan traces
export MOZ_DEBUG_SYMBOLS=1
ac_add_options --enable-debug-symbols
ac_add_options --disable-install-strip

# Settings for an opt build
ac_add_options --enable-optimize="-O2 -gline-tables-only"
ac_add_options --disable-debug

Starting the build process

Now you start the build process using the regular make -f client.mk command.

Starting Firefox

After the build has completed, you can start Firefox from the objdir as usual.

Building only the JavaScript shell

If you want to build only the JavaScript shell instead of doing a full Firefox build, the build script below will probably help you to do so. Before using it, you must of course adjust the path name for LLVM_ROOT to match your setup. Once you have adjusted everything, execute this script in the js/src/ subdirectory and pass a directory name as the first parameter. The build will then be created in a new subdirectory with that name.

#! /bin/sh

if [ -z $1 ] ; then
    echo "usage: $0 <dirname>"
elif [ -d $1 ] ; then
    echo "directory $1 already exists"
else
    autoconf2.13
    mkdir $1
    cd $1
    LLVM_ROOT="/path/to/llvm"
    CC="$LLVM_ROOT/build/bin/clang" \
    CXX="$LLVM_ROOT/build/bin/clang++" \
    CFLAGS="-fsanitize=thread -fPIC -pie" \
    CXXFLAGS="-fsanitize=thread -fPIC -pie" \
    LDFLAGS="-fsanitize=thread -fPIC -pie" \
            ../configure --disable-debug --enable-optimize="-O2 -gline-tables-only" --enable-llvm-hacks --disable-jemalloc
    make -j 8
fi

Using LLVM Symbolizer for faster/better traces

By default, TSan traces are symbolized because otherwise, the runtime suppression list wouldn't work. The default way of symbolizing the traces is much slower than using llvm-symbolizer for that purpose:

LLVM ships with a symbolizer binary that TSan will readily use to immediately output symbolized traces much faster. Furthermore, additional debug information is included in traces when llvm-symbolizer is used. To use it, either make sure the llvm-symbolizer binary is in your PATH or set the environment variable TSAN_OPTIONS="external_symbolizer_path=/path/to/llvm-symbolizer" before running the process. If you've built TSan as described above, then you will want to set the variable to $LLVM_HOME/build/bin/llvm-symbolizer.

Ignoring known races

In some cases, races can be confirmed to be benign and fixing them would cause possible performance issues. TSan offers two possibilities to ignore such known races.

Compile-time blacklisting

Functions with racy memory access can be flagged, such that the compiler will not instrument them. In our codebase, this is done using the MOZ_TSAN_BLACKLIST macro on the function definition. It will expand to the appropriate attributes if the compiler supports them. Compile-time blacklisting is the preferred way of blacklisting, because it does not have a performance impact. Still, blacklisting should be done very carefully so we won't miss important bugs.

Runtime suppressions

To prevent races from showing up at runtime, TSan also provides a runtime suppression list. This list is a good tool to temporarily hide some traces, especially for debugging purposes. However, unlike compile-time blacklisting, runtime suppressions do have a performance impact, especially if the race in question keeps showing up a lot. For more information on how to use the runtime suppression list, see the TSan wiki.