Introduction: Unraveling Android Kernel Panics

Android kernel crashes, often manifesting as sudden device reboots or freezes, are notoriously difficult to debug in the field. Unlike traditional Linux systems with easy access to kernel logs, the constrained and often locked-down nature of Android devices presents unique challenges. However, with the right tools and techniques, it’s possible to reverse engineer these critical failures. This expert-level tutorial delves into setting up and utilizing Kdump for crash dump generation on Android, followed by in-depth analysis using GDB (GNU Debugger) on a host machine. Mastering this workflow is essential for advanced Android developers, security researchers, and anyone performing deep-dive OS customizations.

Understanding Kdump: Your Kernel Crash Reporter

Kdump is a Linux kernel feature that creates a crash dump (vmcore) when the system experiences a kernel panic. It achieves this by booting into a secondary, minimalist kernel (the ‘dump-capture’ kernel) upon a crash, which then captures the memory image of the crashed primary kernel. This vmcore file contains the entire memory state, including registers, stack traces, and data, crucial for post-mortem analysis.

Prerequisites for the Lab

• An Android device with an unlocked bootloader and root access.
• Custom kernel source code matching your device’s running kernel.
• A host PC running Linux (Ubuntu/Debian recommended).
• ADB (Android Debug Bridge) installed and configured.
• Basic familiarity with kernel compilation and GDB.

Step 1: Setting up Kdump on Your Android Device

A. Kernel Configuration

First, you need to ensure your custom kernel is built with Kdump support. Navigate to your kernel source directory and run make menuconfig or modify your .config file directly.

CONFIG_CRASH_DUMP=yCONFIG_PROC_VMCORE=yCONFIG_CRASH_DUMP_EXCLUDE_VMALLOC=y # Optional, reduces vmcore sizeCONFIG_KEXEC=yCONFIG_KEXEC_FILE=y

Recompile your kernel with these options enabled and flash it to your Android device. Ensure you have the corresponding vmlinux (kernel image with debug symbols) readily available on your host PC; this is critical for GDB.

B. Reserving Memory for the Dump-Capture Kernel

Kdump requires a dedicated region of memory for the dump-capture kernel. This is typically done via a bootloader argument. The specific argument varies, but commonly looks like this:

androidboot.kdump_ram=Xs kexec.crash_low_size=Ys

Where Xs is the size for the capture kernel (e.g., 128M) and Ys is the low memory reserve (e.g., 64M). You’ll need to modify your device’s boot image (boot.img) to inject these arguments. Tools like magiskboot or AOSP's mkbootimg can be used for this.

C. Installing the Kdump Kernel

You need to prepare a minimal kernel and an initramfs for the dump-capture kernel. This dump-capture kernel will boot when the primary kernel crashes. The process is complex and often device-specific. Generally, it involves:

1. Compiling a stripped-down kernel.
2. Creating an initramfs that can save the vmcore to persistent storage (e.g., /data/misc/kdump).
3. Using kexec -l to load this kernel into the reserved memory.

For simplicity in a lab environment, many custom ROMs and device trees integrate Kdump setup, sometimes requiring only a command line modification. After configuring and loading, a simple way to test is to force a crash:

adb shellsu -c 'echo c > /proc/sysrq-trigger'

Your device should reboot, and upon restart, you should find a vmcore file (e.g., vmcore.YYYYMMDD-HHMMSS) in the specified directory, typically /data/misc/kdump or /sys/fs/pstore.

Step 2: Preparing Your Host Analysis Environment

A. Install GDB and Debugging Tools

For ARM64 Android devices, you need a multi-arch GDB. On Debian/Ubuntu:

sudo apt updatesudo apt install gdb-multiarch linux-image-$(uname -r)-dbgsym

The linux-image-dbgsym is for your host kernel, not the Android one, but is good practice. Also, ensure you have the crash utility:

sudo apt install crash

B. Obtain Kernel Debug Symbols (vmlinux)

Copy the vmlinux file (the unstripped kernel image with debug symbols) from your kernel build directory to your host PC. This file is crucial for GDB to map addresses to source code and function names.

Step 3: Analyzing the vmcore with GDB

A. Loading the vmcore

Transfer the vmcore file from your Android device to your host PC using ADB:

adb pull /data/misc/kdump/vmcore.YYYYMMDD-HHMMSS .

Now, launch GDB:

gdb-multiarch vmlinux vmcore.YYYYMMDD-HHMMSS

Alternatively, if you’re using the `crash` utility for a quicker initial look:

crash vmlinux vmcore.YYYYMMDD-HHMMSS

The crash utility provides higher-level commands specifically designed for kernel crash analysis (e.g., log, bt, ps, sys, mod).

B. Initial GDB Commands for Crash Analysis

Once GDB is loaded, it will show you the exact crash address. Here are essential commands:

1. Backtrace (bt): The most critical command. It shows the call stack leading up to the crash. This immediately points to the sequence of functions executed.

(gdb) bt

2. Information Registers (info registers or i r): Displays the state of all CPU registers at the time of the crash. Useful for examining argument values, return addresses, and potentially corrupted registers.

(gdb) info registers

3. Disassembly (disas): Disassembles the code around the crash address. Helps understand the instruction that failed.

(gdb) disas $pc

4. Examine Memory (x): Inspects memory content. For example, if bt shows a null pointer dereference from register x0, you can check what x0 contains.

(gdb) x/10i $pc    # Examine 10 instructions at program counter(gdb) x/40wx $sp   # Examine 40 words (hex) on stack pointer

5. List Source Code (list): If vmlinux has debug symbols and you have the kernel source, GDB can show the relevant C code.

(gdb) list *0xffffffc012345678 # List code at specific address

C. Example Analysis Walkthrough

Let’s assume your bt command reveals a crash like this:

#0  0xffffffc012345678 in my_faulty_function (ptr=0x0) at drivers/my_driver/foo.c:123#1  0xffffffc01234abc0 in another_function (data=0x...) at drivers/my_driver/bar.c:456...

This clearly indicates a null pointer dereference in my_faulty_function at line 123 of foo.c. The ptr=0x0 confirms the null pointer. You would then:

1. Use list drivers/my_driver/foo.c:123 to view the problematic code.
2. Examine the call stack (#1, #2, etc.) to understand how my_faulty_function was called and why ptr might have been nullified or not initialized.
3. Check surrounding memory (x command) if it was a memory corruption issue.

Conclusion

Debugging Android kernel crashes is a complex but rewarding skill. By systematically setting up Kdump on your device to capture vmcore files and then using the powerful features of GDB, you gain unparalleled insight into the root causes of kernel panics. This methodology empowers you to diagnose critical system failures, contributing to more stable and robust Android custom kernels and OS modifications. With practice, identifying issues like null pointer dereferences, memory corruption, and race conditions becomes a manageable task, transforming mysterious reboots into actionable debugging insights.