Introduction: The Power of Custom Syscalls in Android

Extending the Android Open Source Project (AOSP) kernel with custom syscalls opens up a realm of possibilities for developers, researchers, and system integrators. A system call (syscall) is the programmatic way in which a computer program requests a service from the kernel of the operating system it is executed on. In the context of Android, this means interacting directly with the core of the operating system, bypassing standard API layers. This detailed guide will walk you through the process of adding a custom syscall to an AOSP kernel specifically tailored for QEMU Android virtual devices, enabling deep system-level customization and research.

Why would you want to add a custom syscall? Common reasons include:

• Implementing new kernel-level features not exposed by existing APIs.
• Improving performance for specific operations by moving logic into the kernel.
• Developing security features or research experiments at a fundamental level.
• Supporting custom hardware interactions directly from the kernel.

This tutorial focuses on a typical x86_64 AOSP build targeting the QEMU emulator, often using the goldfish or ranchu kernels.

Prerequisites and Environment Setup

Before diving into kernel modifications, ensure you have a robust AOSP build environment set up. This typically involves a Linux distribution (Ubuntu LTS is recommended) with ample disk space (200GB+), sufficient RAM (16GB+), and a powerful CPU.

Required Tools and AOSP Source:

• repo tool for managing AOSP source.
• git for version control.
• A fully synchronized AOSP tree (e.g., Android 11 or 12).

To set up your AOSP environment, follow the official AOSP documentation for downloading and building the source. A typical setup would look like this:

mkdir aosp && cd aosp
repo init -u https://android.googlesource.com/platform/manifest -b android-12.0.0_r30
repo sync -j$(nproc)

After syncing, initialize the environment and select a target:

source build/envsetup.sh
lunch aosp_ranchu_x86_64-userdebug

The aosp_ranchu_x86_64-userdebug target is suitable for QEMU x86_64 emulation, leveraging the ranchu kernel.

Step 1: Identifying and Preparing the Kernel Source

The AOSP build system typically uses prebuilt kernels for QEMU. However, for customization, we need to build our own. The kernel sources are usually found within the AOSP tree or fetched separately. For ranchu_x86_64, the kernel is often located under kernel/goldfish or prebuilts/qemu-kernel/. We’ll typically work with a kernel from kernel/goldfish which supports the ranchu architecture.

Navigate to the kernel source directory. For ranchu_x86_64, this might be something like:

cd kernel/goldfish

Ensure your build environment variables are correctly set for the kernel compilation:

export ARCH=x86_64
export CROSS_COMPILE=x86_64-linux-android-
export PATH=$PATH:$(pwd)/../prebuilts/clang/host/linux-x86/clang-r416183b/bin # Adjust clang path

The exact path for CROSS_COMPILE and clang might vary based on your AOSP version. Find the appropriate toolchain under prebuilts/.

Step 2: Defining Your Custom Syscall

Adding a new syscall involves three key steps: choosing a syscall number, adding an entry to the syscall table, and declaring its prototype.

Choosing a Syscall Number

Syscall numbers are architecture-specific. For x86_64, consult the file arch/x86/entry/syscalls/syscall_64.tbl. Look for an unused number at the end of the common ABI list. For this example, let’s assume 335 is free.

Adding to the Syscall Table

Edit arch/x86/entry/syscalls/syscall_64.tbl. Add the following line at the end of the table:

335 common  mysyscall   sys_mysyscall

This line defines syscall number 335 for the common ABI, names it mysyscall for user-space, and maps it to the kernel function sys_mysyscall.

Declaring the Syscall Prototype

The kernel needs to know the function signature of your new syscall. Add its declaration to include/linux/syscalls.h:

asmlinkage long sys_mysyscall(int param1, const char __user *param2);

The asmlinkage keyword is crucial for correct argument passing on the kernel stack. __user indicates that param2 is a pointer to user-space memory, requiring special handling.

Step 3: Implementing the Custom Syscall Function

Now, create the actual implementation of your syscall. For simplicity, we’ll create a new file named kernel/mysyscall.c.

// kernel/mysyscall.c
#include <linux/kernel.h>
#include <linux/syscalls.h>
#include <linux/string.h>
#include <linux/uaccess.h> // For copy_from_user, strncpy_from_user

asmlinkage long sys_mysyscall(int param1, const char __user *param2) {
    char kbuf[256];
    long ret_val = 0;

    // Basic validation for param1
    if (param1 < 0 || param1 > 100) {
        pr_info("mysyscall: Invalid param1 value: %dn", param1);
        return -EINVAL; // Invalid argument
    }

    // Copy string from user space if provided
    if (param2 != NULL) {
        // strncpy_from_user safely copies user string to kernel buffer
        ret_val = strncpy_from_user(kbuf, param2, sizeof(kbuf) - 1);
        if (ret_val < 0) {
            pr_err("mysyscall: Failed to copy string from user space (errno: %ld).n", ret_val);
            return -EFAULT; // Bad address
        }
        kbuf[ret_val] = ''; // Null-terminate the string
        pr_info("mysyscall: Received param1: %d, param2: %sn", param1, kbuf);
    } else {
        pr_info("mysyscall: Received param1: %d, param2: (NULL)n", param1);
    }

    // Return success
    return 0;
}

To ensure this file is compiled, add it to the appropriate Makefile. For kernel/mysyscall.c, you would add the following line to kernel/Makefile:

obj-y += mysyscall.o

Step 4: Compiling the Modified Kernel

Now, compile your customized kernel. Ensure you are in the kernel/goldfish directory (or your specific kernel source directory) and have the correct environment variables set.

make clean
make x86_64_ranchu_defconfig # Or your specific defconfig (e.g., goldfish_defconfig)
make -j$(nproc)

A successful build will produce a kernel image. For x86_64, this is typically arch/x86/boot/bzImage.

Step 5: Integrating the New Kernel into AOSP for QEMU

There are two primary ways to use your custom kernel with QEMU Android:

Option 1: Replacing the Prebuilt Kernel in AOSP

Copy your newly built bzImage to the AOSP prebuilt kernel location:

cp arch/x86/boot/bzImage <AOSP_ROOT>/prebuilts/qemu-kernel/x86_64/bzImage

Then, rebuild your AOSP image (or at least the kernel-related modules) to ensure it uses your new kernel:

cd <AOSP_ROOT>
m -j$(nproc)

Option 2: Specifying the Kernel During Emulator Launch

This is often quicker for testing. Launch the emulator, pointing to your custom kernel image directly:

emulator -kernel /path/to/your/kernel/goldfish/arch/x86/boot/bzImage -avd <YOUR_AVD_NAME> -qemu -append "console=ttyS0 androidboot.console=ttyS0 root=/dev/ram0"

Replace <YOUR_AVD_NAME> with the name of your AOSP-built Android Virtual Device (e.g., aosp_ranchu_x86_64).

Step 6: Developing a User-Space Application to Test the Syscall

To interact with your new syscall, you need a user-space application. We’ll write a simple C program that calls sys_mysyscall. This program needs to be compiled for the Android target architecture (x86_64 in our case).

// test_mysyscall.c
#include <stdio.h>
#include <unistd.h>
#include <sys/syscall.h> // Required for syscall()
#include <errno.h>
#include <string.h>

#define SYS_MYSYSCALL 335 // The syscall number we chose

int main() {
    int ret;
    const char *message = "Hello from Android user space!";

    printf("Calling custom syscall SYS_MYSYSCALL(%d, "%s")n", 42, message);
    ret = syscall(SYS_MYSYSCALL, 42, message);

    if (ret == 0) {
        printf("Syscall returned success.n");
    } else {
        perror("Syscall failed");
        printf("Error number: %d, Message: %sn", errno, strerror(errno));
    }

    printf("nCalling custom syscall with invalid param1 (-5). Expected failure.n");
    ret = syscall(SYS_MYSYSCALL, -5, "Invalid param test");
    if (ret == 0) {
        printf("Syscall returned success (unexpected for invalid param).n");
    } else {
        perror("Syscall failed as expected");
        printf("Error number: %d, Message: %sn", errno, strerror(errno));
    }

    return 0;
}

Compile this program using the Android NDK or AOSP toolchain. Assuming you have the NDK installed, you can compile for x86_64 Android:

export NDK_HOME=<path_to_android_ndk>
$NDK_HOME/toolchains/llvm/prebuilt/linux-x86_64/bin/x86_64-linux-android29-clang -static test_mysyscall.c -o test_mysyscall

Adjust android29 to match your target Android API level. The -static flag is helpful to avoid library issues on the emulator.

Step 7: Executing and Verifying on QEMU Android

Start your AOSP QEMU emulator with the custom kernel (using either Option 1 or 2 from Step 5). Once the emulator is booted and adb is connected:

1. Push your compiled executable to the emulator:

   adb push test_mysyscall /data/local/tmp/

2. Set execute permissions and run the program:

   adb shell "chmod 755 /data/local/tmp/test_mysyscall"
   adb shell "/data/local/tmp/test_mysyscall"

   You should see the output from your user-space application.

3. Verify kernel logs for your syscall’s pr_info messages:

   adb shell dmesg | grep "mysyscall"

   You should see output similar to:

   [ <timestamp>] mysyscall: Received param1: 42, param2: Hello from Android user space!
   [ <timestamp>] mysyscall: Invalid param1 value: -5

Conclusion

You have successfully added a custom syscall to an AOSP kernel, compiled it, integrated it with your QEMU Android emulator, and tested it with a user-space application. This process demonstrates a fundamental way to extend Android’s core functionality. While this example provides a simple