Introduction to AAOS Boot Sequence

The Android Automotive OS (AAOS) boot process, like standard Android, is a complex ballet of interacting components, each critical for a successful system startup. Understanding this sequence is fundamental to effectively diagnose boot failures in custom AOSP builds. The process typically unfolds as follows:

1. Bootloader: Initializes hardware and loads the kernel.
2. Kernel: Initializes system hardware and mounts the root filesystem, then launches the init process.
3. Init: The very first userspace process (PID 1), responsible for parsing .rc scripts, mounting filesystems, spawning services, and setting up the core Android environment.
4. Zygote: Forks new Dalvik/ART virtual machines for Android applications.
5. System Server: Launches core system services like PackageManagerService, ActivityManagerService, etc.
6. Android Framework & Applications: The rest of the Android user experience loads.

Boot failures can occur at any stage, but issues within the init process are particularly common and often manifest as persistent boot loops or hangs, making them challenging to debug without the right approach.

Diagnosing Boot Failures: Initial Triage

When an AAOS device fails to boot, your first priority is to gain access to relevant log data. The method depends on how far the boot process gets:

• Serial Console: This is your most reliable tool for early boot issues (kernel panics, early init failures). Connect a USB-to-serial adapter to the device’s debug port to capture raw kernel and early userspace logs.
• adb logcat: If the device partially boots (e.g., reaches the boot animation, or you can temporarily get into recovery/fastboot) and adbd is active, you can use adb logcat -b all to retrieve all buffered logs.
• dmesg: Provides kernel messages. While often part of the serial console output, it can be run via adb shell dmesg if adb is available.
• ramoops: On some devices, crash dumps are saved to a specific RAM region (/sys/fs/pstore/console-ramoops or similar) that survives reboots, allowing post-mortem analysis of kernel panics or early userspace crashes.

The serial console is paramount. Look for kernel panics (Kernel panic - not syncing), early init errors (lines starting with init:, often marked E/ for error), and SELinux AVC denials.

Deep Dive into the init Process

The init process is driven by configuration files, primarily /system/etc/init/hw/init.rc, /vendor/etc/init/hw/vendor_init.rc, and various .rc files included by them. These files define services, actions, and events that init executes. A common boot loop scenario is when a critical service repeatedly crashes, triggering init‘s respawn mechanism or even a critical_reboot.

Analyzing init Logs for Root Cause

Your primary goal is to identify which service or action is failing. In the serial console output or logcat, look for messages like:

init: Service 'myservice' (pid 1234) exited with status 1

This indicates that a service named ‘myservice’ (with process ID 1234) crashed with exit code 1. A non-zero exit status almost always signifies an error. Follow these steps:

1. Identify the failing service: Pinpoint the exact service name.
2. Examine preceding logs: Often, the actual error message will appear *before* the service exit status, printed by the service itself (e.g., a shared library not found, a configuration file error, a permission issue).
3. Check init.rc definition: Locate the service definition in the relevant .rc file. Pay attention to its command, user/group, capabilities, and dependencies (on triggers, requires).
4. Look for SELinux denials: avc: denied messages are a very common cause of service failures, indicating that a process attempted an operation for which it lacked SELinux permission.

Practical Debugging Steps

Step 1: Get the Logs (Serial Console is King)

Ensure your serial console captures the entire boot process from power-on. This is the only way to catch kernel-level issues and very early init failures.

Step 2: Isolate the Failing Service

Once you suspect a service, you can modify the .rc files (requiring a new build or adb remount on unlocked devices) to:

• Comment out the service: Temporarily remove or comment out the service entry to see if the system boots further. This helps confirm if that service is indeed the blocking factor.
• Use disabled and start manually: Change service myservice ... to service myservice /path/to/binary ... disabled. After booting, if possible, manually start it with adb shell start myservice to get more direct feedback on its failure.
• Use oneshot: For services that should only run once, ensure they have the oneshot keyword. This prevents init from continuously trying to respawn a failing service, which can cause boot loops.

# Example: Temporarily disable a service in init.rc (for testing)init: service problematic_service /vendor/bin/problematic_daemon    class vendor_specific    user system    group system    # disabled # Uncomment to disable for debugging    # oneshot  # Add if it should only run once, not respawn

Step 3: SELinux Troubleshoot

SELinux is a frequent culprit. If you see avc: denied messages in your logs, you have an SELinux policy issue. For debugging purposes *only* (never for production), you can temporarily disable SELinux:

adb shell setenforce 0

If the device boots successfully after disabling SELinux, you’ve confirmed it’s a policy issue. To find the exact policy violation, use audit2allow on your dmesg or audit.log output:

# From your build environment, after pulling relevant logsadb pull /sys/fs/pstore/console-ramoops-0 ./ramoops.logaudit2allow -i ramoops.log -a

This will suggest the necessary SELinux rules to add to your .te files.

Step 4: Dependency Resolution

Ensure that all resources a service depends on are available *before* it starts. This includes:

• Filesystems: Is /vendor mounted? Is /data accessible? Check fstab.target_device for correct mount points and options.
• Devices: Are necessary drivers loaded and device nodes (e.g., /dev/my_device) created?
• Other services: Does your service implicitly or explicitly depend on another service being up and running?

The on property:sys.boot_completed=1 trigger is useful for services that should only start once the entire Android framework is operational.

Advanced Debugging Scenarios

Vendor-Specific init Files (vendor_init.rc)

Many AAOS customizations introduce services and configurations in /vendor/etc/init/hw/vendor_init.rc. Issues here are common. The same debugging principles apply, but remember that the vendor partition is read-only at runtime by default. Modifications require rebuilding or a complex adb remount process (if allowed by your bootloader).

Early Userspace Crashes

Sometimes, crashes occur even before init can fully parse its scripts or before logcat is initialized. In these cases, your only reliable information source is the serial console output and possibly ramoops. Look for:

• Early init parsing errors.
• Kernel OOM (Out Of Memory) killer messages.
• `BUG` or `WARN` messages from the kernel.
• segfault messages from critical userspace binaries launched by init.

Dissecting these requires a deep understanding of the kernel boot process and early userspace execution flow.

Conclusion

Debugging AAOS AOSP boot failures demands a systematic, patient approach. Mastering the use of the serial console, understanding the init process and its configuration files, and being adept at SELinux troubleshooting are critical skills. By methodically isolating failing components and analyzing the rich log data available, even the most stubborn boot issues can be resolved, paving the way for stable and performant custom Android Automotive OS builds.