Introduction: Unraveling Android Kernel Panics with Kdump and GDB

Debugging kernel panics on Android devices can be a daunting task, especially with bespoke kernels and custom hardware. Traditional user-space debugging tools are useless when the kernel itself crashes. This is where Kdump and GDB (GNU Debugger) become indispensable. Kdump is a Linux kernel crash dumping mechanism that captures a system snapshot (vmcore) upon a crash, allowing post-mortem analysis. When combined with a cross-compiled GDB and the crash utility, developers can pinpoint the root cause of elusive kernel panics.

This expert-level guide will walk you through the intricate process of setting up a cross-compilation environment, configuring your Android kernel for Kdump, extracting a kernel core dump, and finally, using GDB and the crash utility for a thorough post-mortem analysis.

Prerequisites and Environment Setup

Before diving in, ensure you have the following:

• Android Kernel Source: The exact source code matching your device’s kernel version.
• Cross-Compilation Toolchain: Typically GCC or Clang for ARM64 (aarch64-linux-android- or similar) or ARM (arm-linux-androideabi-). The Android NDK often provides suitable toolchains.
• crash Utility: A powerful tool for analyzing Linux kernel crash dumps. Compile it from source if your distribution doesn’t have an appropriate version.
• GDB for ARM/ARM64: A cross-debugger capable of understanding your target architecture.
• ADB (Android Debug Bridge): For interacting with your Android device.

Setting Up the Cross-Compilation Environment

First, set up your environment variables for the cross-compiler. Assuming an ARM64 target using a toolchain from the Android NDK:

export ARCH=arm64export CROSS_COMPILE=/path/to/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android-

Configuring Your Android Kernel for Kdump

Kdump requires specific kernel configuration options and memory reservation. You’ll need to modify your kernel’s .config and potentially your device’s Device Tree Source (DTS).

Kernel Configuration (.config)

Navigate to your kernel source directory and run make menuconfig or directly edit .config. Enable the following options:

CONFIG_CRASH_DUMP=yCONFIG_PROC_VMCORE=yCONFIG_DEBUG_INFO=yCONFIG_KALLSYMS=yCONFIG_KALLSYMS_ALL=y

CONFIG_DEBUG_INFO and CONFIG_KALLSYMS are crucial for GDB to resolve symbols and function names within the core dump.

Reserving Memory with Device Tree (DTS/DTB)

Kdump operates by booting a second, smaller kernel into a reserved region of RAM when the primary kernel crashes. This region must be explicitly reserved in the device tree. Locate your device’s DTS file (e.g., arch/arm64/boot/dts/vendor/your_device.dts or .dtsi).

Add a crashkernel entry to your boot arguments. This is typically done in the chosen node:

chosen {    bootargs = "console=ttyS0,115200 root=/dev/mmcblk0pX rw crashkernel=256M";    /* ... other properties ... */};

The crashkernel=256M reserves 256 MB of memory for the dump kernel. The exact size might need tuning based on your device’s total RAM and kernel size. For modern Android devices, 256M or 512M is a common starting point. Rebuild your kernel and DTB:

make defconfig # Or use your existing .configmake Image dtbsmodules

Flash the new Image and dtb.img to your device. The flashing method varies per device (e.g., fastboot, Odin, custom recovery).

Capturing a Kernel Core Dump

Once your Kdump-enabled kernel is running, you can trigger a crash for testing purposes. Be warned: this will reboot your device!

Triggering a Panic

On a rooted Android device, you can write to the trigger file in /proc/sysrq-trigger:

adb shellsu -c 'echo c > /proc/sysrq-trigger'

Alternatively, you can introduce a BUG_ON() macro in your kernel source, rebuild, and flash. For example, in a test module:

#include <linux/kernel.h>#include <linux/module.h>#include <linux/init.h>static int __init crash_test_init(void){    printk(KERN_INFO "Crashing kernel now...");    BUG_ON(1); // Force a kernel panic    return 0;}static void __exit crash_test_exit(void){    // This won't be reached}module_init(crash_test_init);module_exit(crash_test_exit);MODULE_LICENSE("GPL");

After the crash and reboot, the vmcore file should be available on your device. Its location depends on your Kdump configuration, but often it’s mounted under /sys/kernel/debug/crash or directly copied to a persistent storage partition (e.g., /data/misc/kdump/vmcore) by a user-space init script.

Extracting the Core Dump

Use adb pull to retrieve the vmcore file:

adb pull /data/misc/kdump/vmcore .

Debugging with GDB and the crash Utility

With the vmcore file and your compiled vmlinux (the uncompressed kernel image with debug symbols), you’re ready to debug.

Using the crash Utility

The crash utility provides a high-level interface for analyzing kernel dumps. It’s often easier for initial triage than raw GDB.

crash vmlinux vmcore

Once inside the crash prompt, you can use various commands:

• log: Displays the kernel message buffer (dmesg). Look for panic messages.
• bt: Backtrace of the current context (the crashed task).
• ps: List processes.
• mod: List loaded kernel modules.
• dis symbol: Disassemble a function.
• rd address: Read memory at a given address.

For example, to view the backtrace of the crashing process:

crash> btPID: 1234  COMM: 'kworker/u8:0'   TASK: ffff80007d4c0000 (LOCKED)CPU: 0  `#0 [ffff8000787a3ed0] dump_backtrace+0x0/0x2c0#1 [ffff8000787a3f40] show_stack+0x20/0x30#2 [ffff8000787a3fa0] dump_stack_lvl+0x6c/0x9c#3 [ffff8000787a4000] dump_stack+0x18/0x20#4 [ffff8000787a4020] oops_report_fmt+0x11c/0x13c#5 [ffff8000787a4060] __bug_on+0x18/0x24#6 [ffff8000787a4080] crash_test_init+0x18/0x20 [crash_test_module]#7 [ffff8000787a40a0] do_one_initcall+0x54/0x2b0#8 [ffff8000787a40e0] kernel_init_freeable+0x180/0x21c#9 [ffff8000787a4120] kernel_init+0x10/0x118#10 [ffff8000787a4140] ret_from_fork+0x10/0x18`

In this example, the backtrace clearly shows the panic originated from crash_test_init within the crash_test_module, exactly where our BUG_ON(1) was placed.

Using GDB for Deeper Analysis

For more granular control and in-depth analysis, use your cross-compiled GDB. You’ll need the vmlinux file and the vmcore.

aarch64-linux-android-gdb vmlinux

Once GDB starts, load the core dump:

(gdb) target remote | crash --gdb vmlinux vmcore(gdb) bt(gdb) frame 6(gdb) l

The target remote | crash --gdb vmlinux vmcore command pipes the output of crash to GDB, allowing GDB to interpret the core dump. From here, you can use standard GDB commands:

• bt: Display a backtrace.
• info registers: Show CPU registers.
• x/Nx address: Examine memory.
• l *function_name: List source code for a function (if debug info is available).
• print variable_name: Inspect variable values (if symbols are present and in scope).

Advanced Tips and Troubleshooting

• Symbol Resolution: If GDB can’t resolve symbols, ensure CONFIG_DEBUG_INFO and CONFIG_KALLSYMS_ALL are enabled and that you’re using the exact vmlinux file corresponding to the crashed kernel.
• Kdump Boot Issues: If Kdump doesn’t seem to work, check dmesg for messages about crashkernel reservation. Ensure the reserved memory doesn’t conflict with other device resources.
• vmcore Size: Kernel core dumps can be very large. Ensure your device has enough storage if Kdump attempts to copy it to persistent memory before reboot.
• Custom Modules: If your crash occurs in a custom kernel module, ensure you have its debug symbols available and load them into GDB if necessary.

Conclusion

Mastering Kdump and GDB for Android kernel debugging is a critical skill for advanced Android development and embedded systems engineers. This guide provides a robust framework for setting up your environment, capturing core dumps, and performing detailed post-mortem analysis. By systematically applying these techniques, you can effectively diagnose and resolve even the most challenging kernel panics, ensuring the stability and reliability of your bespoke Android systems.