Introduction: The Imperative of Verified Boot in AOSP

Android’s security architecture relies heavily on its boot process, specifically Android Verified Boot (AVB), to ensure the integrity of the operating system from the moment the device powers on. AVB establishes a chain of trust, verifying each stage of the boot process before execution, from the bootloader to the system image. While AOSP provides a robust default implementation, understanding and customizing this verified boot chain is crucial for device manufacturers, security researchers, and advanced developers aiming to integrate proprietary hardware security modules, enforce stricter security policies, or simply gain deeper insights into the Android security model.

This expert-level guide delves into the intricacies of customizing AOSP’s secure boot, walking through the process of generating custom cryptographic keys, modifying the AOSP build system to incorporate these keys, signing boot images, and integrating them into a verifiable boot chain. By the end, you’ll possess the knowledge to build and deploy an AOSP image with your own root of trust.

Understanding Android Verified Boot (AVB) 2.0

AVB 2.0 (also known as dm-verity) is the cornerstone of Android’s integrity protection. It leverages cryptographic signatures to ensure that all executable code and data loaded during boot are precisely what Google or the OEM intended. The boot chain begins with a hardware root of trust (RoT) – typically an immutable key burned into the device’s silicon. This RoT verifies the primary bootloader, which in turn verifies subsequent stages like the secondary bootloader, kernel, ramdisk, and system partitions.

Key Components of AVB:

• Root of Trust (RoT): An immutable public key (or hash thereof) embedded in the device hardware, used to verify the initial bootloader.
• Verification Chain: A cryptographic sequence where each component verifies the next before passing control.
• `avbtool`: The command-line utility used for generating AVB metadata, signing partitions, and manipulating AVB images.
• Hash Trees: Used for protecting large partitions (like `system` and `vendor`) efficiently, allowing for on-demand verification of data blocks.
• Rollback Protection: Ensures that a device cannot boot into an older, potentially vulnerable version of the OS.

Prerequisites for Customization

Before diving into key generation and build system modifications, ensure you have the following:

• AOSP Source Code: A complete synchronized AOSP tree (e.g., Android 12 or newer).
• Development Board: A Pixel device or an AOSP-supported board with an unlocked bootloader (crucial for flashing custom keys).
• Build Environment: A Linux-based build machine with sufficient resources and the necessary AOSP build tools installed.
• Basic Cryptography Knowledge: Understanding of public-key cryptography (RSA), hashing, and digital signatures.

Step 1: Generating Custom AVB Keys

The first step is to create your own set of RSA key pairs. These keys will replace the default AOSP test keys and become your custom root of trust.

mkdir -p ~/android-keyscd ~/android-keysopenssl genrsa -out rsa4096_vbmeta.pem 4096openssl pkcs8 -in rsa4096_vbmeta.pem -topk8 -nocrypt -out rsa4096_vbmeta.pk8avbtool extract_public_key --key rsa4096_vbmeta.pem --output rsa4096_vbmeta.avbpubkey

This sequence generates:

• `rsa4096_vbmeta.pem`: Your private key in PEM format.
• `rsa4096_vbmeta.pk8`: Your private key in PKCS#8 format (required by AOSP).
• `rsa4096_vbmeta.avbpubkey`: The public key in AVB-specific format. This `avbpubkey` file is what you’d ideally flash into the device’s hardware RoT, or at least into the `vbmeta` partition.

Step 2: Modifying AOSP to Use Custom Keys

Now, integrate your custom keys into the AOSP build system. You’ll typically modify your device’s `BoardConfig.mk` and `device.mk` files.

Navigate to your device’s vendor directory (e.g., `device/google/pixel5` or `device//`).

Create a `security` directory within your device configuration:

mkdir -p device///securitycp ~/android-keys/rsa4096_vbmeta.pk8 device///security/cp ~/android-keys/rsa4096_vbmeta.pem device///security/cp ~/android-keys/rsa4096_vbmeta.avbpubkey device///security/

Edit `BoardConfig.mk` for your device:

# In device///BoardConfig.mk...# Replace existing AVB key paths or add these:BOARD_AVB_ENABLE := truedefine BOARD_AVB_SIGN_PARTITIONBOARD_AVB_SIGN_PARTITION_KEY := device///security/rsa4096_vbmeta.pk8BOARD_AVB_SIGN_PARTITION_ALGORITHM := SHA256_RSA4096endefBOARD_AVB_BOOT_KEY_PATH := device///security/rsa4096_vbmeta.avbpubkeyBOARD_AVB_VBMETA_KEY_PATH := device///security/rsa4096_vbmeta.pemBOARD_AVB_VBMETA_ALGORITHM := SHA256_RSA4096BOARD_AVB_RECOVERY_KEY_PATH := device///security/rsa4096_vbmeta.pemBOARD_AVB_SYSTEM_KEY_PATH := device///security/rsa4096_vbmeta.pemBOARD_AVB_VENDOR_KEY_PATH := device///security/rsa4096_vbmeta.pem# Also ensure specific partitions are enabled for AVB verificationBOARD_AVB_BOOT_ADD_HASH_FOOTER_ARGS := --hash_algorithm sha256 --rollback_index 1 --rollback_index_location 0BOARD_AVB_VBMETA_ADD_HASHTREE_FOOTER_ARGS := --hash_algorithm sha256 --rollback_index 1 --rollback_index_location 0BOARD_AVB_SYSTEM_ADD_HASHTREE_FOOTER_ARGS := --hash_algorithm sha256 --rollback_index 1 --rollback_index_location 1BOARD_AVB_VENDOR_ADD_HASHTREE_FOOTER_ARGS := --hash_algorithm sha256 --rollback_index 1 --rollback_index_location 2

The `BOARD_AVB_*_KEY_PATH` variables tell the build system which key to use for signing specific partitions. The `BOARD_AVB_VBMETA_KEY_PATH` is particularly important as `vbmeta.img` itself is signed by this key, and it contains the public keys for verifying other partitions.

Step 3: Building AOSP with Custom Keys

With your custom keys in place and the build configuration updated, you can now build your AOSP image.

source build/envsetup.shlunch <your_device>-userdebug # e.g., aosp_pixel5-userdebugmake -j$(nproc)

The `make` command will take a significant amount of time. During the build process, `avbtool` will be invoked automatically by the AOSP build system to sign all relevant partitions (`boot.img`, `system.img`, `vendor.img`, `vbmeta.img`, etc.) using your specified keys.

Step 4: Flashing Custom Public Key and Signed Images

Once the build completes, you’ll find your signed images in `out/target/product//`. The critical step now is to flash your custom public key onto the device and then flash the signed images. This assumes your bootloader is unlocked.

First, boot your device into `fastboot` mode.

# Erase old vbmeta and flash your custom public keyfastboot erase vbmetafastboot flash --disable-verity --disable-verification vbmeta_a.img # Only for initial setup, disables default verificationfastboot flash avb_custom_key device///security/rsa4096_vbmeta.avbpubkey

Important Note: The `fastboot flash avb_custom_key` command is OEM-specific and may not be available on all devices. On some devices, you might need to use `fastboot flash –set-active=a vbmeta_a out/target/product//vbmeta.img` after flashing, and the device will boot into ‘yellow state’ indicating a custom AVB key. For a true hardware root of trust, the `avbpubkey` would need to be physically fused into the device, which is typically an OEM-only process.

Now, flash all the generated images:

fastboot flash boot out/target/product/<your_device>/boot.imgfastboot flash dtbo out/target/product/<your_device>/dtbo.imgfastboot flash vbmeta out/target/product/<your_device>/vbmeta.imgfastboot flash system out/target/product/<your_device>/system.imgfastboot flash vendor out/target/product/<your_device>/vendor.imgfastboot reboot

Step 5: Verifying the Custom Boot Chain

After the device reboots, you should verify that your custom boot chain is active. Connect your device via ADB and check the verified boot state:

adb shell getprop ro.boot.verifiedbootstate

Expected outputs:

• `green`: Indicates that the device booted successfully, and all images were verified against the original RoT (or your custom RoT, if successfully fused).
• `yellow`: Indicates that the device booted successfully, but a custom AVB key was detected. This is a common state when using `fastboot flash avb_custom_key` on an unlocked bootloader.
• `orange`: Indicates that the device is in an unlocked state and is not performing full verification or is using a custom configuration that is not fully trusted by the original OEM RoT.
• `red`: Indicates a verification failure, meaning the images are corrupted or signed with an unrecognized key.

If you encounter `red` state or boot loops, meticulously review your `BoardConfig.mk` modifications, key paths, and the flashing sequence.

Troubleshooting and Security Considerations

Customizing AOSP Secure Boot is a complex process with potential pitfalls:

• Key Mismatch: Ensure the public key flashed to the device matches the private key used to sign the images.
• Rollback Protection: Be mindful of rollback index values. Flashing an image with a lower rollback index than previously booted can cause a `red` state. Increment the `AVB_ROLLBACK_INDEX` in your `BoardConfig.mk` if you’re making major changes.
• OEM Specifics: The exact `fastboot` commands for flashing custom AVB keys or configuring the RoT can vary significantly between device manufacturers. Refer to your device’s specific documentation or community resources.
• Bootloader State: An unlocked bootloader is a prerequisite for flashing custom keys and signed images. However, an unlocked bootloader inherently reduces security. For production devices, the goal is to relock the bootloader with your custom RoT.

Conclusion

Customizing the AOSP Secure Boot chain is a powerful way to enforce device integrity, integrate specialized hardware, or harden an Android system against tampering. By generating your own cryptographic keys and meticulously integrating them into the AOSP build and flashing process, you establish a unique root of trust for your device. While the process demands careful attention to detail and a solid understanding of both AOSP and cryptographic principles, the ability to control and verify every byte of your Android device’s boot sequence offers unparalleled security and customization capabilities.