Instructions
1.) Get yourself an optimized libxul build of Firefox, with debugger info.
For the Mac, you'll want to read up on Profiling JavaScript with Shark. Here's a sample mozconfig:
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/ff-opt-libxul mk_add_options MOZ_MAKE_FLAGS=-j3 ac_add_options --enable-optimize ac_add_options --enable-libxul ac_add_options --enable-shark ac_add_options --enable-debugger-info-modules ac_add_options --enable-application=browser
2.) You'll want to run Shark on both the browser and [xpcshell], since the host environments differ. Here are some instrumented tests to work from: media:Profiling-ammo.zip
3.) Once you have some changes you'd like to try, you can just rebuild the js/src directory, since it produces its own shared library, even in libxul and static builds.
make -C my-obj-dir/js/src
Other Ways To Profile
For Linux, there are global jprof functions available. See the documentation on the --enable-jprof mozconfig option. We're working to add support for other profilers, such as OProfile, Intel VTune, and Callgrind. When we have JS scriptable profiling options available for all tier 1 platforms, we'll look at adding global start/stop profiling functions.
If you'd like to profile something at a higher level of detail than the JS Shark functions allow, there are corresponding C functions available at the bottom of jsdbgapi.h.
Costs Of Thread Safety
Currently, threadsafe SpiderMonkey costs us 10-15% on some benchmarks vs. a non-threadsafe JS shell. If you're looking to investigate that, build a standalone copy of SpiderMonkey and compare it with xpcshell.