Understanding crash reports

If a user experiences a crash they will be prompted to submit a raw crash report, which is generated by Breakpad. The raw crash report is received by Socorro which creates a processed crash report. The processed crash report is based on the raw crash report but also has a signature, classifications, and a number of improved fields (e.g. OS, product, version). Many of the fields in both the raw crash report and the processed crash report are viewable and searchable on crash-stats. Although there are two distinct crash reports, the raw and the processed, people typically talk about a single "crash report" because crash-stats mostly presents them in a combined way.

Each crash report contains a wealth of data about the crash circumstances. Despite this, many crash reports lack sufficient data for a developer to understand why the crash occurred. As well as providing a general overview, this page aims to highlight parts of a crash report that may provide non-obvious insights.

Note that most crash report fields are visible, but a few privacy-sensitive parts of it are only available to users who are logged in and have "minidump access". A relatively small number of users have minidump access, and they are required to follow certain rules. If you need access, file a bug like this one.

Each crash report has the following tabs: Details, Metadata, Modules, Raw Dump, Extensions, and (optional) Correlations.

Details tab

The Details tab is the first place to look because it contains the most important pieces of information.

Primary fields

The first part of the Details tab shows a table containing the most important crash report fields. It includes such things as when the crash occurred, in which product and version, the crash kind, and various details about the OS and configuration of the machine on which the crash occurred. The following screenshot shows some of these fields.

All fields have a tool-tip. For many fields, the tool-tip describes its meaning. For all fields, the tool-tip indicates the key to use when you want to do searches involving this field. (The field name is usually but not always similar to the search key. E.g. the field "Adapter Device ID" has the search key "adapter_device_id".) These descriptions are shown in the SuperSearchFields API and can be modified by crash-stats superusers.

The fields present in this tab vary depending on the crash kind. Not all fields are always present.

The start of the "Signature" field tells you what kind of crash it is. Some special-purpose annotations are used to indicate particular kinds of crashes.

• Abort: A controlled abort, e.g. via NS_RUNTIMEABORT. (Controlled aborts that occur via MOZ_CRASH or MOZ_RELEASE_ASSERT currently don't get an Abort annotation, but they do get a "MOZ_CRASH Reason" field.)
• OOM | <size>, where <size> is one of large, small, unknown: an out-of-memory (OOM) abort. The <size> annotation is determined by the "OOM Allocation Size" field; if that field is missing <size> will be unknown.
• hang: a hang prior to shutdown.
• shutdownhang: a hang during shutdown.
• IPCError-browser: a problem involving IPC. If the parent Firefox process detects that the child process has sent broken or unprocessable IPDL data, or is not shutting down in a timely manner, it kills the child process with a crash report. These crashes will now have a signature that indicates why the process was killed, rather than the child stack at the moment.

When no special-purpose annotation is present and the signature begins with a stack frame, it's usually a vanilla uncontrolled crash. The crash cause can be determined from the "Crash Reason" field. Most commonly it's a bad memory access. In that case, on Windows you can tell from the reason field if the crash occurred while reading, writing or executing memory (e.g. EXCEPTION_VIOLATION_ACCESS_READ indicates a bad memory read). On Mac and Linux the reason will be SIGSEGV or SIGBUS and you cannot tell from this field what kind of memory access it was.

See this file for a detailed explanation of the crash report signature generation procedure, and for information on how modify this procedure.

There are no fields that uniquely identify the user that a crash report came from, but if you want to know if multiple crashes come from a single user the "Install Time" field is a good choice. Use it in conjunction with other fields that don't change, such as those describing the OS or graphics card, for additional confidence.

For bad memory accesses, the "Crash Address" field can give additional indications what went wrong.

• 0x0 is probably a null pointer deference[*].
• Small addresses like 0x8 can indicate an object access (e.g. this->mFoo) via a null this pointer.
• Addresses like 0xfffffffffd8 might be stack accesses, depending on the platform[*].
• Addresses like 0x80cdefd3 might be heap accesses, depending on the platform.
• Addresses may be poisoned: 0xe4 indicates the address comes from memory that has been allocated by jemalloc but not yet initialized; 0xe5 indicates the address comes from memory freed by jemalloc. The JS engine also has multiple poison values defined in js/src/jsutil.h.

[*] Note that due to the way addressing works on x86-64, if the crash address is 0x0 for a Linux/OS X crash report, or 0xffffffffffffffff for a Windows crash report, it's highly likely that the value is incorrect. (There is a bug report open for this problem.) You can sanity-check these crashes by looking at the raw dump or minidump in the Raw Dump tab (see below).

Some fields, such as "URL" and "Email Address", are privacy-sensitive and are only visible to users with minidump access.

The Windows-only "Total Virtual Memory" field indicates if the Firefox build and OS are 32-bit or 64-bit.

• A value of 2 GiB indicates 32-bit Firefox on 32-bit Windows.
• A value of 3 or 4 GiB indicates 32-bit Firefox on 64-bit Windows (a.k.a. "WoW64"). Such a user could switch to 64-bit Firefox.
• A value much larger than 4 GiB (e.g. 128 TiB) indicates 64-bit Firefox. (The "Build Architecture" field should be "amd64" in this case.)

Some crash reports have a field "ContainsMemoryReport", which indicates that the crash report contains a memory report. This memory report will have been made some time before the crash, at a time when available memory was low. The memory report can be obtained in the Raw Dump tab (see below).

Bug-related information

The second part of the Details tab shows bug-related information, as the following screenshot shows.

The "Report this bug in" links can be used to easily file bug reports. Each one links to a Bugzilla bug report creation page that has various fields pre-filled, such as the crash signature.

The "Related Bugs" section shows related bug reports, as determined by the crash signature.

Stack traces

The third part of the Details tab shows the stack trace and thread number of the crashing thread, as the following screenshot shows.

Each stack frame has a link to the source code, when possible. If a crash is new, the regressing changeset can often be identified by looking for recent changes in the blame annotations for one or more of the top stack frames. Blame annotations are also good for identifying who might know about the code in question.

Sometimes the highlighted source code is puzzling, e.g. the identified line may not touch memory even though the crash is memory-related. This can be caused by compiler optimizations. It's often better to look at the disassembly (e.g. in a minidump) to understand exactly what code is being executed.

Stack frame entries take on a variety of forms.

• The simplest are functions names, such as NS_InitXPCOM2.
• Name/address pairs such as nss3.dll@0x1eb720 are within system libraries.
• Names such as F1398665248_____________________________ ('F' followed by many numbers then many underscores) are in Flash.
• Addresses such as @0xe1a850ac may indicate an address that wasn't part of any legitimate code. If an address such as this occurs in the first stack frame, the crash may be exploitable.

Stack traces for other threads can be viewed by clicking on the small "Show other threads" link.

If the crash report is for a hang, the crashing thread will be the "watchdog" thread, which exists purely to detect hangs; its top stack frame will be something like mozilla::`anonymous namespace'::RunWatchdog. In that case you should look at the other threads' stack traces to determine the problem; many of them will be waiting on some kind of response, as shown by a top stack frame containing a function like NtWaitForSingleObject or ZwWaitForMultipleObjects.

Metadata tab

The Metadata tab is similar to the first part of the Details tab, containing a table with various fields. These are the fields from the raw crash report, ordered alphabetically by field name, but with privacy-sensitive fields shown only to users with minidump access. There is some overlap with the fields shown in the Details tab.

Modules tab

The modules tab shows all the system libraries loaded at the time of the crash, as the following screenshot shows.

On Windows these are mostly DLLs, on Mac they are mostly .dylib files, and on Linux they are mostly .so files.

This information is most useful for Windows crashes, because DLLs loaded by antivirus software or malware often cause Firefox to crash. Correlations between loaded modules and crash signatures can be seen in the "Correlations" tab (see below).

This page says that files lacking version/debug identifier/debug filename are likely to be malware.

Raw Dump tab

The first part of the Raw Dump tab shows the raw crash report, in JSON format. Once again, privacy-sensitive fields are shown only to users with minidump access.

For users with minidump access, the second part of the Raw Dump tab has some links, as the following screenshot shows.

These links are to the following items.

1. A minidump. Minidumps can be extremely useful in understanding a crash report; see this page for an explanation how to use them.
2. The aforementioned JSON raw crash report.
3. The memory report contained within the crash report. Only crash reports with the ContainsMemoryReport field set will have this link.
4. The unredacted crash report, which has additional information.

Extensions tab

The Extensions tab shows which extensions are installed and enabled.

Usually it just shows an ID rather than the proper extension name.

Note that several extensions ship by default with Firefox and so will be present in almost all crash reports. (The exact set of default extensions depends on the release channel.) The least obvious of these has an Id of {972ce4c6-7e08-4474-a285-3208198ce6fd}, which is the default Firefox theme. Some (but not all) of the other extensions shipped by default have the following Ids: [email protected], [email protected], [email protected], [email protected], [email protected].

If an extension only has a hexadecimal identifier, a Google search of that identifier is usually enough to identify the extension's name.

This information is useful because some crashes are caused by extensions. Correlations between extensions and crash signatures can be seen in the "Correlations" tab (see below).

Correlations tab

This tab is only shown when crash-stats identifies correlations between a crash and modules or extensions that are present, which happens occasionally.

See also

• A talk about understanding crash reports, by David Baron, from March 2016.
• A guide to searching crash reports