Building SpiderMonkey with UBSan

1. Compile a recent version of LLVM & Clang.

2. Save the following bash script, fixing LLVM_ROOT to point to your installation.

#! /bin/sh

if [ -z $1 ] ; then
    echo "usage: $0 <dirname>"
elif [ -d $1 ] ; then
    echo "directory $1 already exists"
else
    autoconf2.13
    autoconf213
    mkdir $1
    cd $1
    LLVM_ROOT="$HOME/llvm"
    SANFLAG="-fsanitize=undefined -fno-sanitize=alignment,float-cast-overflow,float-divide-by-zero,vptr -Dxmalloc=myxmalloc" \
    CC="$LLVM_ROOT/build/Release+Asserts/bin/clang" \
    CXX="$LLVM_ROOT/build/Release+Asserts/bin/clang++" \
    CFLAGS="$SANFLAG" \
    CXXFLAGS="$SANFLAG" \
    MOZ_LLVM_HACKS=1 \
            ../configure --enable-debug --disable-optimize
    make -j 8
fi

3. Use the script to compile SpiderMonkey.

This enables all the cheap undefined behavior checks other than:

• alignment, which hits known bugs in SpiderMonkey, and is more implementation-defined (slow on x86 / crash on ARM) than undefined behavior
• float-cast-overflow, which hits known bugs in SpiderMonkey, and isn't exploited by today's compilers
• float-divide-by-zero, which Jesse doesn't think is actually undefined behavior (aside from the question of whether CPU overflow flags are set)
• vptr, a check that requires RTTI, which is disabled by default in SpiderMonkey

4. When you hit a bug and want a stack trace, run under gdb with a breakpoint at the end of __ubsan::Diag::~Diag().  The stack trace should show a function such as __ubsan_handle_load_invalid_value or __ubsan_handle_type_mismatch being called by the buggy C++ code.  (For automated testing outside of gdb, you can instead build with -fsanitize=undefined-trap -fsanitize-undefined-trap-on-error, but then you lose UBSan's diagnostics and the ability to continue past errors.)

Known bugs. Please file new bugs with e.g. [-fsanitize=float-cast-overflow] in the status whiteboard.