Introduction to Advanced Ftrace for Android Kernel Debugging

Debugging complex issues within the Android kernel or userspace often requires more than standard logging or `strace`. When you need to understand precise execution flows, function call arguments, or timing characteristics deep within the system, Linux’s built-in `ftrace` framework, especially when augmented with dynamic tracing capabilities like `kprobes` and `uprobes`, becomes indispensable. This expert-level guide will walk you through leveraging these powerful tools on Android to gain unprecedented visibility into both kernel and userspace events.

Ftrace provides a rich set of tracers and event hooks. While basic event tracing is powerful, `kprobes` and `uprobes` unlock dynamic instrumentation, allowing you to attach trace points to virtually any function in the kernel or a userspace binary without recompiling. This is particularly crucial in a production-like Android environment where custom kernel builds might not always be feasible or desirable for quick debugging.

Setting Up Your Android Ftrace Environment

Before diving into `kprobes` and `uprobes`, ensure your Android device is rooted and has a kernel compiled with `ftrace`, `kprobes`, and `uprobes` support. Most modern Android kernels enable these by default. You’ll need `adb` access to the device shell.

First, verify `ftrace` is available:

adb shell
su
ls /sys/kernel/debug/tracing

If the directory exists, you’re good to go. All `ftrace` operations will be performed by writing to or reading from files within this directory. Remember to always run `su` to gain root privileges.

Ftrace Basics Revisited

A quick refresher on basic `ftrace` commands:

• Enable a tracer: echo function > /sys/kernel/debug/tracing/current_tracer
• Enable/disable tracing: echo 1 > /sys/kernel/debug/tracing/tracing_on / echo 0 > /sys/kernel/debug/tracing/tracing_on
• Clear trace buffer: echo > /sys/kernel/debug/tracing/trace
• Read trace output: cat /sys/kernel/debug/tracing/trace

For `kprobes` and `uprobes`, we typically use the `nop` tracer or the `events` mechanism directly without a specific `current_tracer` set, relying on the `kprobe_events` and `uprobe_events` files to manage probe definitions.

Unleashing `kprobes` for Kernel-Level Events

`kprobes` allow you to dynamically insert breakpoints into the Linux kernel and execute custom trace actions when these breakpoints are hit. This is incredibly powerful for observing kernel function calls, their arguments, and return values without recompiling the kernel.

Defining a `kprobe` Event

To define a `kprobe`, you write its definition to /sys/kernel/debug/tracing/kprobe_events. The general syntax is p:<event_group>/<event_name> <function_name>[:<offset>] <arguments>. To trace a function’s return, use `r:`. Common arguments include `$comm` (current process name), `$pid` (process ID), `arg1`, `arg2`, etc. for function parameters.

Let’s trace calls to the kernel’s `sys_openat` function, which is fundamental for file system operations. We’ll capture the process name, PID, and the filename being opened (first argument).

adb shell
su
echo 'p:openat/my_sys_openat sys_openat dfd=%ax filename=%si flags=%dx mode=%r10' > /sys/kernel/debug/tracing/kprobe_events
echo 1 > /sys/kernel/debug/tracing/events/openat/my_sys_openat/enable
echo 1 > /sys/kernel/debug/tracing/tracing_on

# Now, perform some file operations on the device, e.g., 'ls /'

echo 0 > /sys/kernel/debug/tracing/tracing_on
cat /sys/kernel/debug/tracing/trace

# Clean up the probe
echo '-:openat/my_sys_openat' > /sys/kernel/debug/tracing/kprobe_events

In the `p:openat/my_sys_openat` definition:

• `p:` signifies a kprobe (for entry point).
• `openat/my_sys_openat` is the event group and event name.
• `sys_openat` is the kernel function to probe.
• `dfd=%ax filename=%si flags=%dx mode=%r10` captures arguments. On ARM64, registers used for arguments are typically `x0`, `x1`, `x2`, etc. for the first few arguments. For example, `%x0` for `dfd`, `%x1` for `filename`, `%x2` for `flags`, `%x3` for `mode`. You might need to consult architecture-specific calling conventions or kernel source. For illustration, we use generic x86-like registers here, but for ARM64, they would be `x0`, `x1`, `x2`, `x3`.

Reading `kprobe` Output

The output in `trace` will show lines prefixed with `my_sys_openat`, providing details on each `sys_openat` call, including the process, PID, and the arguments we specified.

Diving into `uprobes` for Userspace Insights

`uprobes` extend this dynamic tracing capability to userspace applications and shared libraries. This is invaluable for understanding how specific userspace functions are called, what arguments they receive, and their execution flow within an application context.

Identifying Userspace Function Offsets

Unlike kernel functions, userspace functions require specifying the exact binary path and either the function name or, more reliably, its offset within the binary. You can find offsets using `nm` or `readelf` on the target binary.

For example, let’s trace `malloc` in `libc.so`. First, locate `libc.so` and find the offset of `malloc`:

adb shell
su
LIB_PATH=$(find /apex/com.android.runtime/javalib/arm64 /system/lib64 -name libc.so 2>/dev/null | head -n 1) # Adjust path as needed
if [ -z "$LIB_PATH" ]; then LIB_PATH=$(find /system/lib64 /vendor/lib64 -name libc.so 2>/dev/null | head -n 1); fi

if [ -f "$LIB_PATH" ]; then
  echo "Found libc.so at: $LIB_PATH"
  MALLOC_OFFSET=$(readelf -s $LIB_PATH | grep ' malloc$' | awk '{print $2}')
  echo "malloc offset: 0x$MALLOC_OFFSET"
else
  echo "libc.so not found! Please check paths."
fi

Note: The exact path to `libc.so` can vary between Android versions and devices. The above snippet tries common locations.

Defining a `uprobe` Event

With the path and offset, define the `uprobe` to `uprobe_events`. The syntax is similar to `kprobes`: `p:<event_group>/<event_name> <binary_path>[:<offset>] <arguments>`. For ARM64, userspace function arguments are typically passed in registers `x0`, `x1`, `x2`, `x3`, etc., similar to the kernel.

adb shell
su
# Assuming LIB_PATH and MALLOC_OFFSET were found previously

# Example with actual offset from the previous step
# MALLOC_OFFSET_DECIMAL=$(printf "%d" "0x$MALLOC_OFFSET") # Convert hex to decimal
# If you're using a specific function name, you don't need the offset

echo "p:libc/my_malloc $LIB_PATH:malloc size=%x0" > /sys/kernel/debug/tracing/uprobe_events
echo 1 > /sys/kernel/debug/tracing/events/libc/my_malloc/enable
echo 1 > /sys/kernel/debug/tracing/tracing_on

# Now, launch an application or perform an action that triggers malloc, e.g., 'toybox ls'

echo 0 > /sys/kernel/debug/tracing/tracing_on
cat /sys/kernel/debug/tracing/trace

# Clean up the probe
echo '-:libc/my_malloc' > /sys/kernel/debug/tracing/uprobe_events

This will show every call to `malloc` within any process that uses that `libc.so`, along with the requested allocation size (`%x0`).

Combining `kprobes` and `uprobes` for Holistic Tracing

The true power emerges when you combine both types of probes. Imagine tracing a userspace application’s call to `open()`, then immediately seeing the corresponding `sys_openat` kernel call, and further observing how the kernel handles the VFS layer. This allows for end-to-end tracing, bridging the userspace-kernel boundary, which is invaluable for diagnosing complex performance bottlenecks or security vulnerabilities.

For instance, you could trace `open` in `libc.so` and `sys_openat` in the kernel simultaneously to see the exact path an `open` call takes from an app into the kernel’s file system handler.

Advanced Considerations and Best Practices

Performance Overhead

While dynamic tracing is powerful, `kprobes` and `uprobes` do introduce overhead. Each probe adds a small execution cost. Excessive probes, especially in high-frequency paths, can significantly impact system performance. Always enable tracing only for the duration needed and remove probes promptly.

Filtering Trace Output

The trace output can be voluminous. Utilize `trace_filter` and `event` filters to narrow down the data:

# Filter kernel trace events by process name
echo 'common_comm == "my_app_process"' > /sys/kernel/debug/tracing/events/openat/my_sys_openat/filter

# Filter userspace trace events by argument value (e.g., malloc size > 1024)
echo 'size > 1024' > /sys/kernel/debug/tracing/events/libc/my_malloc/filter

Cleaning Up Probes

Always remember to disable events and remove probes when done:

echo 0 > /sys/kernel/debug/tracing/tracing_on
echo 0 > /sys/kernel/debug/tracing/events/<group>/<event_name>/enable
echo '-:<group>/<event_name>' > /sys/kernel/debug/tracing/kprobe_events # or uprobe_events

Or, to clear all probes:

echo > /sys/kernel/debug/tracing/kprobe_events
echo > /sys/kernel/debug/tracing/uprobe_events

Security Implications

Dynamic tracing is a privileged operation. Improper use or leaving probes active could expose sensitive system information. Always exercise caution and only use these tools in controlled debugging environments.

Conclusion

Ftrace, coupled with `kprobes` and `uprobes`, transforms into an incredibly versatile and powerful debugging suite for Android. It provides the granularity and flexibility needed to diagnose even the most elusive issues residing deep within the kernel or critical userspace components. By mastering these tools, you gain an expert-level capability to observe, understand, and ultimately resolve complex system behaviors, elevating your Android debugging prowess significantly.