Introduction: The Promise and Perils of OverlayFS on Android

OverlayFS has become a cornerstone technology for modern Linux distributions and, increasingly, for Android devices. It allows for the creation of an “overlay” filesystem, combining a read-only lower layer (like your immutable /system or /vendor partitions) with a writable upper layer. This design is invaluable for custom ROMs, system modifications, or development environments where changes need to be ephemeral or applied on top of an immutable base without altering it directly. However, ensuring modifications persist across reboots can be a significant challenge for even experienced Android modders.

This expert-level guide delves deep into diagnosing and fixing common persistence errors encountered when working with OverlayFS on Android. We’ll cover the intricacies of its operation, typical pitfalls, and provide actionable steps and commands to help you achieve reliable, persistent customizations.

Understanding OverlayFS Fundamentals on Android

At its core, OverlayFS merges multiple filesystems. On Android, this typically involves:

• Lower Directory (lowerdir): The read-only base filesystem, e.g., /system_root, /system, or /vendor partitions.
• Upper Directory (upperdir): A writable directory, usually located on the /data partition, where all changes (new files, modifications, deletions) are stored.
• Work Directory (workdir): A temporary, empty directory also on the same filesystem as the upperdir, used internally by OverlayFS for atomic operations. It must be empty when the mount is performed.
• Merged Directory: The final, merged view presented to the user, combining the lowerdir and upperdir.

The challenge for persistence lies in correctly configuring and managing the upperdir and ensuring the OverlayFS mount is initiated correctly at boot time.

Common Persistence Errors and Their Symptoms

1. Incorrect Mount Options

OverlayFS is sensitive to mount options. Incorrect or missing options can lead to unexpected behavior, including non-persistence or even mount failures.

• Missing index=off or uuid/gid issues: Some kernels require index=off, especially if the underlying filesystem doesn’t fully support all OverlayFS features. Android often uses specific UID/GID mappings, and issues can arise if these are not handled correctly.
• Filesystem type mismatches: Ensure the underlying filesystems for upperdir and workdir (usually ext4 or f2fs on /data) are compatible.

2. Filesystem Permissions and Ownership Issues

The upperdir and workdir must be accessible and writable by the user or process initiating the OverlayFS mount (typically root or an init process).

• Incorrect permissions: If upperdir or workdir have restrictive permissions (e.g., 0700 owned by an unprivileged user), OverlayFS won’t be able to write to them.
• Incorrect ownership: Similar to permissions, incorrect ownership can block write access.

3. Incorrect Directory Structure or State

OverlayFS has strict requirements for the state of its directories at mount time.

• Non-empty workdir: The workdir MUST be empty when OverlayFS is mounted. If it contains data, the mount will fail.
• Existing data in upperdir from previous failed mounts: Sometimes a failed mount can leave residual data in the upperdir, causing issues with subsequent mounts.

4. SELinux Context Mismatches

Android’s robust SELinux policies are a frequent cause of persistence problems. If the upperdir or its contents have incorrect SELinux contexts, writes will be denied.

• Incorrect file_contexts for upperdir: The directory where upperdir resides might not have the correct SELinux type, preventing OverlayFS from operating within its boundaries.
• Mismatched contexts of overlaid files: New files created in the OverlayFS might not inherit or be assigned the correct SELinux contexts, leading to access denied errors for system processes trying to use them.

5. Kernel Support and Configuration

Older Android kernels or custom kernels might lack full or correctly configured OverlayFS support.

• Missing kernel modules: While less common on modern Android, some features might require specific kernel modules.
• Kernel version limitations: Very old kernels (pre-3.18) might have limited or buggy OverlayFS implementations.

6. Boot-time Race Conditions or init Script Issues

The timing and execution environment of OverlayFS mounts are critical during Android’s boot process.

• Mounting too early/late: If the /data partition isn’t fully decrypted or available, or if the lowerdir isn’t ready, the mount will fail.
• Errors in fstab or init.rc: Typos, incorrect paths, or logical errors in the boot scripts responsible for mounting OverlayFS.

Diagnosing OverlayFS Persistence Errors

Effective diagnosis involves checking various system logs and states.

Step 1: Verify Mount Status and Options

After booting, check the system’s active mounts.

adb shellmount | grep overlay

Look for your intended OverlayFS mount point. Examine the options carefully. A typical successful mount might look like:

overlay on /mnt/merged_root type overlay (rw,relatime,lowerdir=/system_root,upperdir=/data/overlay/upper,workdir=/data/overlay/work)

If your mount isn’t listed or shows incorrect options, this is your first clue.

Step 2: Inspect Kernel Logs (dmesg and logcat)

The kernel is usually vocal about mount failures.

adb shell dmesg | grep -i overlayadb shell logcat -b kernel | grep -i overlay

Look for error messages like