Refcount tracing and balancing are advanced techniques for tracking down leak of refcounted objects found with BloatView. The first step is to run Firefox with refcount tracing enabled, which produces one or more log files. Refcount tracing logs calls to Addref
and Release
, preferably for a particular set of classes, including call-stacks in symbolic form (on platforms that support this). Refcount balancing is a follow-up step that analyzes the resulting log to help a developer figure out where refcounting went wrong.
How to build for refcount tracing
Build with --enable-debug
or --enable-logrefcnt
.
How to run with refcount tracing on
There are several environment variables that can be used.
First, you select one of three environment variables to choose what kind of logging you want. You almost certainly want XPCOM_MEM_REFCNT_LOG
.
-
XPCOM_MEM_REFCNT_LOG
-
Setting this environment variable enables refcount tracing. If you set this environment variable to the name of a file, the log will be output to that file. You can also set it to 1 to log to stdout or 2 to log to stderr, but these logs are large and expensive to capture, so you probably don't want to do that. WARNING: you should never use this without
XPCOM_MEM_LOG_CLASSES
and/orXPCOM_MEM_LOG_OBJECTS
, because without some filtering the logging will be completely useless due to how slow the browser will run and how large the logs it produces will be.
-
XPCOM_MEM_COMPTR_LOG
-
This environment variable enables logging of additions and releases of objects into
nsCOMPtr
s. This requires C++ dynamic casts, and is probably not what you want to do.
-
XPCOM_MEM_ALLOC_LOG
-
For platforms that don't have stack-crawl support, XPCOM supports logging at the call site to
AddRef
/Release
using the usual cpp__FILE__
and __LINE__ number macro expansion hackery. This results in slower code, but at least you get some data about where the leaks might be occurring from.
You must also set one or two additional environment variables, XPCOM_MEM_LOG_CLASSES
and XPCOM_MEM_LOG_OBJECTS,
to reduce the set of objects being logged, in order to improve performance to something vaguely tolerable.
-
XPCOM_MEM_LOG_CLASSES
- This variable should contain a comma-separated list of names which will be used to compare against the types of the objects being logged. For example:
env XPCOM_MEM_LOG_CLASSES=nsDocShell XPCOM_MEM_REFCNT_LOG=./refcounts.log ./mach run
-
This will log the
AddRef
andRelease
calls only for instances ofnsDocShell
while running the browser usingmach
, to a filerefcounts.log
. Note that settingXPCOM_MEM_LOG_CLASSES
will also list the serial number of each object that leaked in the "bloat log" (that is, the file specified by theXPCOM_MEM_BLOAT_LOG
variable; see the BloatView documentation for more details). An object's serial number is simply a unique number, starting at one, that is assigned to the object when it is allocated.
You may use an object's serial number with the following variable to further restrict the reference count tracing:
-
XPCOM_MEM_LOG_OBJECTS
-
Set this variable to a comma-separated list of object serial number or ranges of serial number, e.g.,
1,37-42,73,165
(serial numbers start from 1, not 0). When this is set, along withXPCOM_MEM_LOG_CLASSES
andXPCOM_MEM_REFCNT_LOG
, a stack track will be generated for only the specific objects that you list. For example,
env XPCOM_MEM_LOG_CLASSES=nsDocShell XPCOM_MEM_LOG_OBJECTS=2 XPCOM_MEM_REFCNT_LOG=./refcounts.log ./mach run
-
will log stack traces to
refcounts.log
for the 2ndnsDocShell
object that gets allocated, and nothing else.
Post-processing step 1: finding the leakers
First you have to figure out which objects leaked. The script tools/rb/find-leakers.pl
does this. It grovels through the log file, and figures out which objects got allocated (it knows because they were just allocated because they got AddRef()
-ed and their refcount became 1). It adds them to a list. When it finds an object that got freed (it knows because its refcount goes to 0), it removes it from the list. Anything left over is leaked.
The scripts output looks like the following.
0x00253ab0 (1) 0x00253ae0 (2) 0x00253bd0 (4)
The number in parentheses indicates the order in which it was allocated, if you care. Pick one of these pointers for use with Step 2.
Post-processing step 2: building the balance tree
Now that you've picked an object that leaked, you can build a balance tree. This process takes all the stack AddRef()
and Release()
stack traces and munges them into a call graph. Each node in the graph represents a call site. Each call site has a balance factor, which is positive if more AddRef()
s than Release()
es have happened at the site, zero if the number of AddRef()
s and Release()
es are equal, and negative if more Release()
es than AddRef()
s have happened at the site.
To build the balance tree, run tools/rb/make-tree.pl
, specifying the object of interest. For example:
perl -w make-tree.pl --object 0x00253ab0 < my-leak.log
This will build an indented tree that looks something like this (except probably a lot larger and leafier):
.root: bal=1 main: bal=1 DoSomethingWithFooAndReturnItToo: bal=2 NS_NewFoo: bal=1
Let's pretend in our toy example that NS_NewFoo()
is a factory method that makes a new foo and returns it. DoSomethingWithFooAndReturnItToo()
is a method that munges the foo before returning it to main()
, the main program.
What this little tree is telling you is that you leak one refcount overall on object 0x00253ab0
. But, more specifically, it shows you that:
NS_NewFoo()
"leaks" a refcount. This is probably "okay" because it's a factory method that creates anAddRef()
-ed object.DoSomethingWithFooAndReturnItToo()
leaks two refcounts. Hmm...this probably isn't okay, especially because...main()
is back down to leaking one refcount.
So from this, we can deduce that main()
is correctly releasing the refcount that it got on the object returned from DoSomethingWithFooAndReturnItToo()
, so the leak must be somewhere in that function.
So now say we go fix the leak in DoSomethingWithFooAndReturnItToo()
, re-run our trace, grovel through the log by hand to find the object that corresponds to 0x00253ab0
in the new run, and run make-tree.pl
. What we'd hope to see is a tree that looks like:
.root: bal=0 main: bal=0 DoSomethingWithFooAndReturnItToo: bal=1 NS_NewFoo: bal=1
That is, NS_NewFoo()
"leaks" a single reference count; this leak is "inherited" by DoSomethingWithFooAndReturnItToo()
; but is finally balanced by a Release()
in main()
.
Hints
Clearly, this is an iterative and analytical process. Here are some tricks that make it easier.
Ignore balanced trees. The make-tree.pl
script accepts an option --ignore-balanced
, which tells it not to bother printing out the children of a node whose balance factor is zero. This can help remove some of the clutter from an otherwise noisy tree.
Play Mah Jongg. An unbalanced tree is not necessarily an evil thing. More likely, it indicates that one AddRef()
is cancelled by another Release()
somewhere else in the code. So the game is to try to match them with one another.
Exclude Functions. To aid in this process, you can create an "excludes file", that lists the name of functions that you want to exclude from the tree building process (presumably because you've matched them). make-tree.pl
has an option --exclude [file]
, where [file]
is a newline-separated list of function names that will be excluded from consideration while building the tree. Specifically, any call stack that contains that call site will not contribute to the computation of balance factors in the tree.
How to instrument your objects for refcount tracing and balancing
The process is the same as instrumenting them for BloatView, because BloatView and refcount tracing share underlying infrastructure.