Introduction to Android Things Custom OS Porting

Android Things, Google’s platform for Internet of Things (IoT) devices, offers a robust framework for building smart, connected products. While it provides a streamlined development experience for application developers, custom OS porting to unique hardware platforms presents a different set of challenges. This process, often involving deep dives into kernel configurations, device trees, and hardware abstraction layers (HALs), can quickly turn into a debugging nightmare. This guide aims to shed light on common issues encountered during Android Things custom OS porting and provide expert-level troubleshooting strategies to help you navigate these complexities.

The Labyrinth of Custom OS Build and Setup

The journey begins with setting up the build environment and successfully compiling the Android Things OS for your target hardware. Even at this initial stage, pitfalls are numerous.

Initial Setup and Build Environment Woes

Before you even touch a device, ensuring your build environment is correctly configured is crucial. Common issues include:

• Incorrect Repo Sync: Failing to initialize or synchronize the AOSP repository correctly can lead to missing source files or outdated components.
• Missing Build Dependencies: Linux distributions often require specific packages (e.g., `openjdk`, `git`, `make`, `python`) for AOSP compilation.
• Wrong Build Target (`lunch` command): Selecting an incorrect `lunch` target will result in a build tailored for a different device, leading to compatibility issues down the line.

Troubleshooting Steps:

1. Verify `repo` status: Ensure all projects are synchronized and no local modifications are pending.

   repo status

2. Install all required packages as per the AOSP build environment setup documentation.
3. Double-check your `lunch` selection against your device’s AOSP product name. For custom boards, this might be `aosp_-userdebug`.

Compilation Failures

Even with a perfect setup, compilation can fail due to various reasons:

• Out-of-Memory Errors: Building AOSP requires significant RAM and swap space.
• Toolchain Mismatches: Incorrect versions of `gcc`, `clang`, or `binutils`.
• Syntax Errors in Custom Patches: If you’ve applied custom kernel or HAL patches, ensure they are syntactically correct and compatible with the AOSP version.

Troubleshooting Steps:

1. Increase swap space and ensure sufficient RAM.
2. Use the recommended `make` command with appropriate job count (e.g., `make -j$(nproc)` or `make -j8`).
3. Carefully review compilation logs for error messages, often indicating file paths and line numbers. Focus on the first error encountered, as subsequent errors might be a cascade.

Decoding Boot Failures and Boot Loops

Successfully compiling the OS is only half the battle. The next hurdle is getting your custom OS to boot on your hardware.

The Dreaded Boot Loop

A device repeatedly restarting after displaying the boot logo (or nothing at all) is a classic symptom of a severe underlying issue. Causes often include:

• Kernel Panic: The Linux kernel encounters a fatal error and cannot proceed.
• Incorrect Device Tree Blob (DTB): The DTB specifies hardware configuration to the kernel. Errors here can lead to drivers failing to initialize or incorrect resource allocation.
• Corrupted `init.rc` or `fstab`: Essential system initialization scripts or file system tables can cause boot failures if malformed.
• Missing Critical Drivers: If fundamental drivers (e.g., for storage, memory controller) are missing or fail to load, the system won’t boot.

Troubleshooting Steps:

1. Serial Console/UART: This is your most critical tool. Connect a serial console to your board’s UART port to capture early boot logs (`dmesg`). This will show kernel messages, driver loading status, and often the exact point of failure.

   sudo picocom -b 115200 /dev/ttyUSB0

   (Replace `/dev/ttyUSB0` with your serial port)

2. Test Minimal System: Try flashing only the `boot.img` and `system.img` with a known good configuration (e.g., from an officially supported board, if available for comparison).
3. DTB Verification: Use `dtc -I dtb -O dts -o output.dts your_dtb_file.dtb` to decompile your DTB and visually inspect it for errors against your hardware specifications.
4. Analyze `init.rc` and `fstab`: Ensure these files are syntactically correct and point to valid partitions/paths.

Black Screen on Boot

The device powers on, but nothing appears on the display. This usually points to display-related issues.

• Display Driver Issues: The graphics driver might not be initializing correctly.
• Incorrect Panel Timings/Configuration in DTB: The DTB defines display parameters like resolution, refresh rate, and interface (HDMI, MIPI DSI, LVDS).
• Hardware Wiring Issues: Physical connection problems between the SoC and the display panel.

Troubleshooting Steps:

1. Check `dmesg` (via serial console): Look for messages related to display initialization (`drm`, `panel`, `i915`, `msm`). Any error here is a strong indicator.
2. Verify DTB Display Nodes: Ensure the display controller, panel, and connector nodes in your DTB accurately reflect your hardware. Pay close attention to timing parameters, clock rates, and power sequences.
3. Test with a Known Good Display: If possible, try a different display or interface to rule out external display issues.

Peripheral Connectivity and Driver Mayhem

Even if the OS boots, getting all peripherals to function correctly is another common hurdle.

USB/GPIO Not Functioning

When USB devices aren’t detected or GPIO pins don’t respond:

• Missing Kernel Modules: The necessary kernel drivers for your USB host controller or GPIO controller might not be compiled or loaded.
• Incorrect Pin Muxing: GPIO pins often have multiple functions (GPIO, UART, I2C, etc.). Incorrectly configuring the pin multiplexer will prevent the pins from operating as intended.
• Device Tree Overlays: For more modular configurations, incorrect device tree overlays (DTOs) can cause issues.

Troubleshooting Steps:

1. Inspect `dmesg` and `logcat`: Search for messages related to your USB host controller (e.g., `xhci`, `ehci`) or GPIO controller.
2. Check `lsmod`: Verify that relevant kernel modules are loaded. If not, ensure they are enabled in your kernel configuration (`.config`) and compiled.
3. Verify Device Tree Entries: Examine the DTB for correct USB controller nodes, `pinctrl` (pin controller) configurations, and GPIO definitions. Ensure interrupt lines are correctly defined.

   adb shell cat /sys/kernel/debug/pinctrl/pinctrl-maps

   This command (if `debugfs` is mounted and `pinctrl` debugging is enabled) can show current pin configurations.

4. Test Pin Muxing: Manually toggle GPIOs via `sysfs` (e.g., `/sys/class/gpio`) to confirm basic functionality if the driver loads.

Wi-Fi and Bluetooth Woes

Networking connectivity is paramount for IoT devices. Issues here can be frustrating.

• Incorrect Firmware: Wi-Fi/Bluetooth modules often require proprietary firmware files. If these are missing or incorrect, the module won’t initialize.
• Missing/Incorrect Drivers: Similar to other peripherals, the kernel driver for your specific Wi-Fi/Bluetooth chip is essential.
• Regulatory Domain Issues: Incorrect country code settings can prevent certain channels or power levels from being used.

Troubleshooting Steps:

1. Firmware Presence: Ensure the correct firmware files are present in `/vendor/etc/firmware` (or `/system/etc/firmware`) on your device. Check `dmesg` for firmware loading attempts and failures.
2. Check `dmesg` for Driver Initialization: Look for messages from your Wi-Fi (`wlan`) and Bluetooth (`bluetooth`, `hci`) drivers.
3. Use `iwconfig` and `hciconfig` (via `adb shell`): These tools can show the status of your wireless interfaces. If `wlan0` or `hci0` don’t appear, the driver or module hasn’t initialized correctly.

   adb shell iwconfigadb shell hciconfig

4. Verify Kernel Configuration: Ensure `CONFIG_WLAN`, `CONFIG_BT`, and specific driver options for your chip are enabled.

Power Management and Stability Challenges

A stable and power-efficient device is critical for IoT applications.

Excessive Power Consumption/Thermal Issues

If your device runs hot or drains battery quickly:

• Unoptimized Kernel: Generic kernel configurations might not utilize platform-specific power-saving features.
• Misconfigured Power States: Incorrect idle states (C-states, P-states) for CPU and peripherals.
• Constant Polling: Drivers that continuously poll hardware instead of using interrupts.

Troubleshooting Steps:

1. Use `dumpsys batterystats`: On an Android system, this provides a wealth of information about power usage by components.
2. Monitor CPU Frequencies and Governors: Use `adb shell cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq` to check current CPU frequency and `/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` for the active governor. Ensure governors like `ondemand` or `interactive` are configured correctly.
3. Inspect Kernel Configuration: Enable platform-specific power management features (e.g., suspend-to-RAM, deep sleep states).

Random Reboots/Crashes

Unexpected reboots or system freezes are often hard to diagnose.

• Kernel Instabilities: Bugs in custom kernel patches or driver code can lead to panics.
• Memory Corruption: Bad pointers or incorrect memory management can cause crashes.
• Watchdog Timer: A misconfigured watchdog timer can reboot the system if it detects a freeze.

Troubleshooting Steps:

1. Analyze `logcat` and `dmesg` (especially after a reboot): Look for `FATAL EXCEPTION`, `kernel panic`, `OOM` (Out Of Memory), or watchdog messages.
2. Memory Stress Tests: Run memory-intensive applications or tools to expose potential memory issues.
3. Disable Watchdog (temporarily): If you suspect the watchdog, try disabling it in the kernel configuration or boot arguments during debugging.

Essential Debugging Tools and Strategies

Mastering these tools and approaches is key to success:

Serial Console is Your Best Friend

Always have a serial console connected during early development. It provides invaluable low-level debug output that ADB cannot. It’s the only way to see kernel panics, early boot errors, and U-Boot/bootloader messages.

ADB and Fastboot Mastery

Beyond basic `adb shell` and `fastboot flash`, learn to use:

• `adb logcat -b all`: View all system logs.
• `adb pull /data/anr/traces.txt`: Retrieve ANR traces.
• `fastboot boot `: Temporarily boot a kernel without flashing it, useful for quick tests.

Kernel Debugging (kgdb/ftrace)

For deep kernel issues, consider setting up `kgdb` for source-level kernel debugging or `ftrace` for tracing kernel function calls. These require significant setup but provide unparalleled insight.

Incremental Changes and Version Control

Always make small, incremental changes. Test each change thoroughly. Use a robust version control system (like Git) to track all modifications, allowing you to easily revert to a stable state when issues arise.

Conclusion

Porting Android Things to custom hardware is a challenging but rewarding endeavor. It demands a deep understanding of embedded Linux, Android internals, and careful hardware-software integration. By systematically approaching issues with the right tools—primarily the serial console, ADB, and a methodical troubleshooting mindset—you can overcome the common debugging nightmares. Patience, persistence, and a strong cup of coffee are your most reliable companions on this journey.