Introduction: Understanding Out-of-Tree Kernel Patches

Developing custom kernels for Android devices often involves modifying the Linux kernel source code beyond the official releases from device manufacturers or upstream Linux. These modifications, when developed externally to the main kernel repository or a specific vendor branch, are often referred to as “out-of-tree patches.” This comprehensive guide will walk you through the process of setting up your build environment, developing a sample patch, applying it, compiling a custom kernel, and finally flashing it onto an Android device.

While the term “out-of-tree module” specifically refers to a loadable kernel module compiled separately, “out-of-tree patch” in this context means a set of changes (e.g., a `.patch` file) that you generate and apply to an existing kernel source tree. These changes become integrated into the kernel source before compilation, effectively making them “in-tree” for your custom build, but they originate “out-of-tree” from the perspective of the official project branches.

Prerequisites and Environment Setup

Before diving into kernel development, ensure you have the necessary tools and kernel source code. This guide assumes a Linux-based development environment.

1. Obtain the Kernel Source Code

You’ll need the exact kernel source code matching your device’s existing kernel version. Often, device manufacturers or custom ROM communities (like LineageOS) provide these. For a generic example, let’s assume you’re working with a Qualcomm-based device and have cloned a common kernel tree:

git clone <KERNEL_SOURCE_REPOSITORY_URL> android_kernel/vendor/device_name
cd android_kernel/vendor/device_name
git branch -a # Identify relevant branches
git checkout <YOUR_DEVICE_KERNEL_BRANCH>

Replace `<KERNEL_SOURCE_REPOSITORY_URL>` and `<YOUR_DEVICE_KERNEL_BRANCH>` with the actual values for your device.

2. Set Up the Cross-Compilation Toolchain

Android kernels are typically compiled for ARM or ARM64 architectures. You’ll need a suitable cross-compilation toolchain, such as `aarch64-linux-android-` for ARM64 or `arm-linux-gnueabi-` for ARM. Google provides these as part of the Android NDK or prebuilt toolchains. It’s often easier to use prebuilt GCC or Clang toolchains provided by kernel developers (e.g., Proton Clang, AOSP Clang).

# Example for AARCH64 GCC 7.x (adjust path as needed)
export ARCH=arm64
export SUBARCH=arm64
export KBUILD_ARCH=arm64
export CROSS_COMPILE=<PATH_TO_YOUR_TOOLCHAIN>/bin/aarch64-linux-android-
export PATH=<PATH_TO_YOUR_TOOLCHAIN>/bin:$PATH

Make sure to replace `<PATH_TO_YOUR_TOOLCHAIN>` with the actual path to your toolchain’s `bin` directory.

3. Configure the Kernel

Each device has a specific kernel configuration. You’ll need to use your device’s `defconfig` file.

make <YOUR_DEVICE_DEFCONFIG>

Example: `make vendor/msm-perf_defconfig` or `make lineageos_X_defconfig`. If you don’t know it, check your kernel’s `arch/arm64/configs/` or `arch/arm/configs/` directory.

Developing an Out-of-Tree Patch

Now, let’s create a simple patch. For demonstration, we’ll modify a hypothetical driver to add a `printk` message. Assume we want to add a debug message to a driver located at `drivers/misc/example_driver.c`.

1. Make Your Modifications

Edit the target file. For instance, open `drivers/misc/example_driver.c` and add a `printk` call within an initialization function:

// drivers/misc/example_driver.c

static int __init example_driver_init(void)
{
    printk(KERN_INFO "Example Driver: Initializing custom functionality!n");
    // ... existing code ...
    return 0;
}

static void __exit example_driver_exit(void)
{
    printk(KERN_INFO "Example Driver: Exiting custom functionality!n");
    // ... existing code ...
}

module_init(example_driver_init);
module_exit(example_driver_exit);

Save the file after making your changes.

2. Generate the Patch File

Use `git diff` to generate the patch. Ensure your changes are not yet committed.

git diff > ../../my_custom_patch.patch

This command creates a `my_custom_patch.patch` file in the parent directory of your kernel source, capturing only your uncommitted changes.

Applying the Patch and Building the Kernel

1. Apply the Patch

Before applying, you might want to revert your working tree to a clean state or switch to a fresh clone if you want to keep your original modifications separate. For applying to a clean tree:

# Ensure you are in the kernel source root directory
# If you just generated the patch, you might need to revert your changes first
git reset --hard # DANGER: This discards all uncommitted changes!

patch -p1 < ../../my_custom_patch.patch

The `-p1` option tells `patch` to strip one leading directory component from file names in the patch file. This is common when patches are generated from the root of the source tree.

2. Build the Kernel and Device Tree Blob (DTB)

After applying the patch, compile the kernel. Android devices typically use `Image.gz` or `Image.gz-dtb` and a separate `dtb` (Device Tree Blob).

# Clean previous builds (optional but recommended)
make clean
make mrproper

# Configure (if not done already or if you made Kconfig changes)
make <YOUR_DEVICE_DEFCONFIG>

# Build the kernel image and modules
make -j$(nproc) O=out

# The output will typically be in O=out/arch/arm64/boot/ or O=out/arch/arm/boot/
# Look for Image.gz-dtb or Image.gz and dtb

`$(nproc)` uses all available CPU cores for faster compilation. The `O=out` flag directs build artifacts to an `out` directory, keeping the source tree clean.

3. Repack the Android Boot Image

Android devices boot from a `boot.img` file, which contains the kernel (`Image.gz-dtb`), ramdisk, and device tree (if not merged into `Image.gz`). You’ll need to extract your device’s existing `boot.img` to get its ramdisk and then repack it with your new kernel.

Extracting `boot.img`

First, get your device’s `boot.img`. You can pull it from a factory image or dump it directly from the device (requires root):

adb pull /dev/block/by-name/boot boot.img # Requires root

# Or from a factory image: unzip <FACTORY_IMAGE>.zip && mv boot.img .

Use a tool like `ai_bootimg_tools` or `magiskboot` to extract the components:

./magiskboot unpack boot.img

This will typically extract `kernel`, `ramdisk.cpio.gz`, and sometimes `dtb` or `dtb.img`.

Repacking `boot.img`

Replace the extracted `kernel` file with your newly compiled `Image.gz-dtb` (or `Image.gz` and `dtb` separately). Then, repack:

cp out/arch/arm64/boot/Image.gz-dtb kernel # Assuming Image.gz-dtb is your kernel
# If dtb is separate: cp out/arch/arm64/boot/dtb.img dtb

./magiskboot repack boot.img
# Or, using mkbootimg (more involved with arguments):
mkbootimg --kernel kernel --ramdisk ramdisk.cpio.gz --base <BOOT_BASE_ADDRESS> --pagesize <PAGE_SIZE> --cmdline "<KERNEL_COMMAND_LINE>" --board "<BOARD_NAME>" -o new_boot.img

You’ll need to know the correct base address, page size, command line, and board name. These can often be found by inspecting your original `boot.img` with `magiskboot` or `unpack_bootimg.py` scripts.

Flashing and Testing

1. Flash the New Boot Image

Reboot your Android device into fastboot mode.

adb reboot bootloader

Once in fastboot, flash your `new_boot.img`:

fastboot flash boot new_boot.img
fastboot reboot

2. Verify Your Changes

After the device reboots, you can check for your `printk` messages using `dmesg` or `logcat` (with root access).

adb shell
su
dmesg | grep "Example Driver"

You should see your custom debug message in the kernel logs.

Troubleshooting Common Issues

• Bootloops: This is the most common issue. Double-check your kernel configuration, toolchain setup, and ensure the patch was applied correctly. A minor syntax error or incorrect Kconfig option can cause a bootloop.
• `make` errors: Carefully read the error messages. Missing header files, incorrect toolchain paths, or syntax errors in your C code are typical culprits.
• Device doesn’t boot into fastboot/recovery: If flashing fails or leads to a hard brick, you might need to restore a working `boot.img` via a custom recovery (if accessible) or re-flash a full factory image.
• Patch conflicts: If `patch -p1` fails, it means the patch cannot be applied cleanly. This often happens if the kernel source tree is not the exact version the patch was created against, or if conflicting changes exist. You’ll need to manually resolve the conflicts or rebase your patch.

Conclusion

Developing and applying out-of-tree kernel patches for Android devices is a powerful technique for custom ROM developers and advanced users to tailor their device’s behavior. While it requires a deep understanding of the Linux kernel and specific device intricacies, mastering this workflow opens up possibilities for custom features, performance optimizations, and debugging at the deepest level of the operating system. Always proceed with caution, back up your device, and ensure you have a recovery plan in case of issues.