Introduction: Unlocking Deep Kernel Insights with Ftrace

Ftrace, the Function Tracer, stands as an indispensable tool within the Linux kernel, offering unparalleled visibility into the kernel’s runtime behavior. For Android developers and kernel engineers, Ftrace provides a window into performance bottlenecks, race conditions, and intricate system interactions that are often elusive with user-space debugging tools. While Ftrace offers a rich set of built-in events, including function calls, scheduling events, and I/O operations, complex debugging scenarios frequently demand more bespoke solutions. This is where custom Ftrace plugins become invaluable: they allow you to instrument specific kernel code paths with tailor-made tracing events, providing the exact data needed for pinpoint diagnosis.

Standard Ftrace events, though powerful, might not capture the precise context, argument values, or return states of a specific function crucial to your investigation. Crafting a custom Ftrace plugin – essentially, a kernel module that registers its own trace events – empowers you to extend Ftrace’s capabilities, transforming it from a general-purpose observer into a highly specialized diagnostic instrument.

The Ftrace Architecture and Custom Tracepoints

At its core, Ftrace operates through the tracefs pseudo-filesystem, typically mounted at /sys/kernel/debug/tracing. This filesystem provides a programmatic interface to enable/disable tracers, configure events, and read trace data. Custom events integrate seamlessly into this architecture, appearing as new directories and files within tracefs.

Custom trace events are primarily defined using the TRACE_EVENT macro within a kernel module. This macro allows you to:

• Define the event’s name and its event category.
• Specify the arguments the event will record (TP_PROTO, TP_ARGS).
• Describe the structure of the data that will be stored in the trace buffer (TP_STRUCT__entry).
• Provide a human-readable format string for the event (TP_printk).

By registering these custom tracepoints, you create hooks that your kernel module can trigger at specific points in the kernel code, capturing custom data payloads into the Ftrace buffer.

Setting Up Your Android Kernel Development Environment

Before diving into code, ensure you have a proper development environment:

1. Android Kernel Source: Obtain the kernel source code for your target Android device (e.g., from AOSP or your device manufacturer).
2. Cross-Compilation Toolchain: A suitable ARM/ARM64 GCC/Clang toolchain is required to build the kernel module (e.g., from AOSP prebuilts or Linaro).
3. ADB: Android Debug Bridge for pushing modules and interacting with the device shell.
4. Build System: A configured kernel build environment to compile external modules.

Developing a Simple Custom Ftrace Event Module

Let’s create a kernel module that defines a simple custom tracepoint named my_custom_event within the my_events category. This event will record a custom integer value and a string.

1. Create `trace_events.h`

This header defines our custom tracepoint. It must be included once with CREATE_TRACE_POINTS and once with TRACE_INCLUDE_FILE.

#undef TRACE_SYSTEM
#define TRACE_SYSTEM my_events

#if !defined(_MY_EVENTS_TRACE_H) || defined(TRACE_HEADER_MULTI_READ)
#define _MY_EVENTS_TRACE_H

#include <linux/tracepoint.h>

TRACE_EVENT(my_custom_event,
	TP_PROTO(int my_val, const char *my_str),
	TP_ARGS(my_val, my_str),

	TP_STRUCT__entry(
		__field(int, val)
		__field(const char *, str)
	),

	TP_fast_assign(
		__entry->val = my_val;
		__entry->str = my_str;
	),

	TP_printk("val=%d str=%s", __entry->val, __entry->str)
);

#endif /* _MY_EVENTS_TRACE_H */

/* This part must be outside the header guard */
#include <trace/define_trace.h>

2. Create `my_custom_module.c`

This is the kernel module that will register and trigger the event.

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/delay.h>
#include <trace/events/my_events.h> // Include our generated trace header

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Your Name");
MODULE_DESCRIPTION("A custom Ftrace event module");

static int __init my_module_init(void)
{
    int i;
    printk(KERN_INFO "my_custom_module: Initializingn");
    
    // Trigger the custom tracepoint several times
    for (i = 0; i < 5; i++) {
        trace_my_custom_event(i * 10, "Hello from custom Ftrace");
        msleep(100);
    }
    
    printk(KERN_INFO "my_custom_module: Custom events triggeredn");
    return 0;
}

static void __exit my_module_exit(void)
{
    printk(KERN_INFO "my_custom_module: Exitingn");
}

module_init(my_module_init);
module_exit(my_module_exit);

3. Create `Makefile`

A simple Makefile to compile the module against your kernel source.

obj-m := my_custom_module.o

my_custom_module-objs := my_custom_module.o trace_events.o

KDIR := /path/to/your/android/kernel/source

all:
	$(MAKE) -C $(KDIR) M=$(PWD) modules

clean:
	$(MAKE) -C $(KDIR) M=$(PWD) clean

Replace `/path/to/your/android/kernel/source` with the actual path to your kernel source directory.

Building, Deploying, and Tracing

1. Build the module

Navigate to your module directory and run make.

cd /path/to/my_custom_module
make

This will generate `my_custom_module.ko` and `trace_events.ko` (though `trace_events.ko` is typically not loaded directly, it’s used during the build process to generate the necessary tracepoint definitions).

2. Deploy to Android Device

Push the compiled kernel module to your rooted Android device:

adb push my_custom_module.ko /data/local/tmp/
adb shell
su
insmod /data/local/tmp/my_custom_module.ko

3. Enable and Read Custom Events

Once the module is loaded, your custom event should appear in tracefs. Now, let’s enable it and read the trace data.

# On the Android device shell (as root)
cd /sys/kernel/debug/tracing/

# List available events to confirm your event exists
cat available_events | grep my_events
# Expected output: my_events:my_custom_event

# Enable the custom event
echo 1 > events/my_events/my_custom_event/enable

# Clear previous traces (optional but good practice)
echo > trace

# Now, the custom module will have already triggered events upon insmod.
# If you want to trigger more, you'd call a kernel function that uses trace_my_custom_event.
# For this example, the events are triggered during init.

# Read the trace output
cat trace

You should see output similar to:

# tracer: nop
#
# entries-in-buffer/entries-written: 5/5   #P:1
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU# |||||  TIMESTAMP  FUNCTION
#              | |       | |||||     |         |
      insmod-123   [001] d.... 1234.567890: my_custom_event: val=0 str=Hello from custom Ftrace
      insmod-123   [001] d.... 1234.667890: my_custom_event: val=10 str=Hello from custom Ftrace
      insmod-123   [001] d.... 1234.767890: my_custom_event: val=20 str=Hello from custom Ftrace
      insmod-123   [001] d.... 1234.867890: my_custom_event: val=30 str=Hello from custom Ftrace
      insmod-123   [001] d.... 1234.967890: my_custom_event: val=40 str=Hello from custom Ftrace

4. Disable and Unload

echo 0 > events/my_events/my_custom_event/enable
rmmod my_custom_module

Real-World Application: Tracing Specific Function Arguments

The true power emerges when you use custom tracepoints to instrument existing kernel functions without modifying their source directly (using kprobes) or when you need to embed specific debugging information directly into your own new kernel features. For instance, imagine you suspect an issue in a particular kernel function, say `some_driver_read_data()`, and you need to see the `offset` and `size` arguments it receives, along with its return value.

You could add `trace_some_driver_event(offset, size, ret_val)` directly into `some_driver_read_data()` and at its return points. This would give you a precise, lightweight logging mechanism that’s superior to `printk` because it doesn’t incur the same performance overhead and is easily controlled via tracefs.

Advanced Considerations

• Performance: While Ftrace is optimized, adding many highly frequent tracepoints can still introduce overhead. Be judicious.
• Data Types: Be mindful of passing complex data structures to `TP_ARGS`. Often, it’s better to pass pointers and dereference them within `TP_fast_assign` to copy relevant fields to `__entry`’s structure.
• String Handling: For strings, `TP_fast_assign` often copies the pointer. If the string buffer can go out of scope, you might need `strncpy` to copy the content to a buffer within `TP_STRUCT__entry`.
• Integration with existing tools: The generated trace data can be further processed by tools like `trace-cmd`, `KernelShark`, or custom Python scripts for advanced analysis and visualization.

Conclusion

Crafting custom Ftrace plugins represents a significant leap in your Android kernel debugging capabilities. By defining and triggering your own specialized trace events, you gain the ability to capture highly specific, contextual data that standard tracing mechanisms might miss. This empowers you to diagnose intricate issues with greater precision, optimize performance more effectively, and ultimately gain a deeper understanding of the Android kernel’s inner workings. Mastering this technique transforms Ftrace into a truly bespoke debugging companion, tailored to your exact investigative needs.