Debugging B2G using gdb and related tools

Starting the debugger in single-process mode

To restart Firefox OS and run it under gdb control, simply use the run-gdb.sh script:

./run-gdb.sh

If Firefox OS is already running and you want to attach to it without restarting it, you can do so like this:

./run-gdb.sh attach

Debugging out-of-process tasks

Because of the threaded nature of Firefox OS, you often need to be able to debug tasks other than the main B2G task. To do this, the simplest way is to use the b2g-ps command to find out the PID of the process that you need to debug:

$ adb shell b2g-ps
b2g              root      106   1     189828 56956 ffffffff 40101330 S /system/b2g/b2g
Browser          app_0     4308  106   52688  16188 ffffffff 400db330 S /system/b2g/plugin-container

Here, Browser is the child process used as the "content process" for the browser application. So if you want to debug the content process, in this example, do:

$ ./run-gdb.sh attach 4308

Sometimes, it is useful to be notified immediately of any child process creation. This can be achieved by starting run-gdb.sh with the MOZ_DEBUG_CHILD_PROCESS environment variable:

MOZ_DEBUG_CHILD_PROCESS=1 ./run-gdb.sh

Having done this, launching an OOP application on Firefox OS will output the PID of the plugin-container for the new task, and will sleep for 30 seconds, enough time for you to run the attach command we saw above:

$ ./run-gdb.sh attach 4308

If you are trying to debug something that occurs during boot, you have to launch the debugger instance for the new application fairly quickly. Once the new debugger is launched, you should immediately press "c" to continue running the new task.

Debugging core files

By default, Firefox OS will not dump core files when a process crashes. You may enable them on debug builds with the following:

$ adb shell setprop persist.debug.coredump all
$ adb reboot

A reboot is required for the change to take effect and will cause all processes to core dump if they crash once B2G has begun to initialize. For platforms using an older kernel (< 3.0, such as hamachi), or if you only desire core dumps from B2G-specific processes (i.e. b2g, plugin-container) and not Gonk, you may enable them with the following:

$ adb shell setprop persist.debug.coredump b2g
$ adb reboot

Core files are saved to /data/core. You may open a core file produced by b2g with the following:

$ adb pull /data/core .
$ ./run-gdb.sh core b2g.1286.1412337385.core

You may open a content process core file with the following:

$ adb pull /data/core .
$ ./run-gdb.sh core plugin-container Camera.1329.1412617644.core

Support

What level of functionality to expect

The following debugging features at least should definitely work. If they don't, it's likely that a simple tweak to your setup will make them work:

• Symbols for all libraries (except perhaps certain drivers on certain Android phones)
• Backtraces with full debug info (except for optimized-away argument values)
• Breakpoints: you should be able to break on a symbol, or on a file:line, or on an address. All should work.
• Single-stepping ('s' and 'n' should both work)

The following debugging features are not supported. Don't try to use them.

• Watchpoints.

Troubleshooting

Here are a few things to try first whenever GDB is not working as described above.

Ensure that your B2G clone is up-to-date

Always keep in mind to that to update your B2G clone you must run these two commands:

git pull
./repo sync

Forgetting the git pull there is a typical reason why you'd end up with an old run-gdb.sh and not benefit from recent improvements.

Ensure that you are attaching to the right process

Attaching to the wrong process (e.g. main B2G process versus Browser process) would explain why your breakpoints don't get hit.

Ensure that symbols are correctly read

1. In gdb, use info shared to check that symbols are correctly read:

   (gdb) info shared
   From        To          Syms Read   Shared Object Library
   0xb0001000  0xb0006928  Yes         out/target/product/otoro/symbols/system/bin/linker
   0x40051100  0x4007ed74  Yes         /hack/b2g/B2G/out/target/product/otoro/symbols/system/lib/libc.so
   0x401ab934  0x401aba2c  Yes         /hack/b2g/B2G/out/target/product/otoro/symbols/system/lib/libstdc++.so
   ...

2. The Syms Read column should say Yes everywhere. Maybe on some android phone you would see Yes (*) for some system libraries or drivers; that would be OK. You should not see any No.
3. If you do see a No, that is your first problem and you must solve it before looking at anything else.
4. Look for any error messages in the terminal output just after you typed your run-gdb.sh command.
5. Also check in that terminal output that the GDB command is sane. In particular, its last command line argument should be the path to your b2g executable. Here is a sane example:

   prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/arm-linux-androideabi-gdb -x /tmp/b2g.gdbinit.bjacob /hack/b2g/B2G/objdir-gecko/dist/bin/b2g

6. Check the value of these GDB variables: solib-search-path and solib-absolute-prefix:

   (gdb) show solib-search-path
   The search path for loading non-absolute shared library symbol files is /hack/b2g/B2G/objdir-gecko/dist/bin:out/target/product/otoro/symbols/system/lib:out/target/product/otoro/symbols/system/lib/hw:out/target/product/otoro/symbols/system/lib/egl:out/target/product/otoro/symbols/system/bin:out/target/product/otoro/system/lib:out/target/product/otoro/system/lib/egl:out/target/product/otoro/system/lib/hw:out/target/product/otoro/system/vendor/lib:out/target/product/otoro/system/vendor/lib/hw:out/target/product/otoro/system/vendor/lib/egl.
   (gdb) show solib-absolute-prefix
   The current system root is "out/target/product/otoro/symbols".