Introduction

Android applications often employ various obfuscation techniques to protect their intellectual property and prevent reverse engineering. Among the more sophisticated methods is the use of custom class loaders. Instead of relying on the system’s default PathClassLoader or DexClassLoader for the primary classes.dex, an application might load an initial, minimal DEX file that then dynamically loads and decrypts subsequent, obfuscated DEX files from internal storage, assets, or even network resources. This case study details a practical approach to identify and bypass such a custom class loader implementation in a commercial Android application.

The Challenge: Identifying Custom Class Loader Obfuscation

The first sign of custom class loader obfuscation is often a suspiciously small or empty classes.dex file after unpacking the APK. When decompiled, this primary DEX might contain only a few classes, typically related to the application’s entry point (e.g., a custom Application class) or a simple splash screen. The actual business logic and core components are conspicuously absent. This indicates that the real application code is being loaded at runtime.

Typical Symptoms:

• classes.dex is unusually small or contains only loader-related code.
• Absence of expected application package structures or core classes in initial decompilation.
• Presence of encrypted or unusually named binary files (e.g., payload.dat, app.bin) in the assets/ or res/raw/ directories.
• Calls to java.io.File operations, javax.crypto packages, or dynamic class loading mechanisms like dalvik.system.DexClassLoader or java.lang.ClassLoader.loadClass() in the initial code.

Step 1: Initial APK Analysis and Decompilation

Our journey begins with standard static analysis tools. We’ll use apktool to unpack the APK and Jadx-GUI for initial decompilation.

First, unpack the APK:

apktool d com.example.obfuscatedapp.apk -o obfuscated_app_unpacked

Navigate into the unpacked directory and inspect the smali/ folder. You’ll likely find a minimal set of SMALI files. Open the original APK with Jadx-GUI. Look for the application’s entry point, usually defined in the AndroidManifest.xml. This is typically a custom Application class or an early activity’s onCreate() or attachBaseContext() method. In our case, the AndroidManifest.xml pointed to a custom MyApplication class:

<application android:name=".MyApplication" ...>

Decompiling MyApplication.java revealed a method similar to this:

protected void attachBaseContext(Context base) {    super.attachBaseContext(base);    try {        File cacheDir = getDir("dex_cache", MODE_PRIVATE);        File decryptedDexFile = new File(cacheDir, "payload.dex");        if (!decryptedDexFile.exists()) {            byte[] encryptedBytes = readEncryptedAsset(this, "payload.bin");            byte[] decryptedBytes = decrypt(encryptedBytes, "mysecretkey");            FileOutputStream fos = new FileOutputStream(decryptedDexFile);            fos.write(decryptedBytes);            fos.close();        }        DexClassLoader dcl = new DexClassLoader(            decryptedDexFile.getAbsolutePath(),            cacheDir.getAbsolutePath(),            null,            getClassLoader()        );        // Replace the current ClassLoader with the new one        Object currentActivityThread = Reflector.invokeStaticMethod("android.app.ActivityThread", "currentActivityThread");        Field mPackages = Reflector.getField("android.app.ActivityThread", "mPackages");        Map<String, WeakReference<LoadedApk>> packages = (Map<String, WeakReference<LoadedApk>>) mPackages.get(currentActivityThread);        WeakReference<LoadedApk> wr = packages.get(getPackageName());        LoadedApk loadedApk = wr.get();        Field mClassLoader = Reflector.getField("android.app.LoadedApk", "mClassLoader");        mClassLoader.set(loadedApk, dcl);    } catch (Exception e) {        e.printStackTrace();    }}

Step 2: Locating the Encrypted DEX File(s)

From the decompiled code, it’s clear the app is looking for payload.bin in assets, decrypting it, and writing it to dex_cache/payload.dex before loading it. We found payload.bin in the assets/ directory of the unpacked APK. This binary blob is the encrypted core of the application.

Step 3: Bypassing the Class Loader – Dynamic Analysis with Frida

Since the application decrypts and loads the DEX file at runtime, the most effective way to obtain the unencrypted DEX is to dump it from memory. Frida, a dynamic instrumentation toolkit, is ideal for this task.

Frida Setup:

• Rooted Android device or emulator.
• Frida server running on the device.
• Frida client on your host machine.

We’ll write a Frida script to hook the DexClassLoader constructor or, more precisely, the FileOutputStream.write() method that writes the decrypted DEX file to disk. Intercepting the write() operation allows us to dump the decrypted DEX bytes before the class loader even gets a chance to load it.

Here’s a Frida script that targets the FileOutputStream.write method:

Java.perform(function() {    var FileOutputStream = Java.use('java.io.FileOutputStream');    FileOutputStream.write.overload('[B').implementation = function(bArr) {        // Check if the file being written is our target payload.dex        var filePath = this.$handle.getFd().getFilePath(); // Simplified, actual path retrieval might be more complex        if (filePath && filePath.endsWith('payload.dex')) {            console.log('Intercepted write to payload.dex! Dumping content...');            var fileName = '/data/local/tmp/dumped_payload.dex';            var file = new File(fileName, 'wb');            file.write(bArr);            file.close();            console.log('DEX dumped to: ' + fileName);        }        return this.write.overload('[B]').call(this, bArr);    };    console.log('Frida script loaded: Hooking FileOutputStream.write to catch payload.dex');});

To run this script:

frida -U -f com.example.obfuscatedapp -l frida_dex_dump.js --no-pause

The --no-pause flag ensures the app starts immediately, allowing our hook to be active from the beginning. Once the app launches and performs the decryption, the script will execute, and you should see output indicating the DEX file has been dumped. Use adb pull to retrieve it from your device:

adb pull /data/local/tmp/dumped_payload.dex .

Step 4: Decompiling the Dumped DEX Files

Now that we have dumped_payload.dex, we can open it with Jadx-GUI. This time, you should see the full application logic, including all the packages, classes, and methods that were previously hidden. You can now analyze the core functionality, identify sensitive code, or proceed with further reverse engineering steps.

Step 5: Advanced Techniques (Briefly)

While dynamic analysis with Frida is powerful, other methods exist for more complex scenarios:

• Static Patching: If the decryption key or algorithm is easily identifiable, you could patch the APK’s SMALI code to directly dump the decrypted bytes or even disable the obfuscation altogether.
• Emulator Snapshots: For highly dynamic loaders, taking emulator snapshots after the DEX is loaded into memory can be useful for analysis with tools like Volatility or other memory forensics frameworks.
• Custom Android Runtime: Modifying the Android Runtime (ART) source code to log or dump DEX files during class loading can provide a very low-level approach, though it’s significantly more involved.

Conclusion

Cracking custom class loader obfuscation requires a combination of static and dynamic analysis. By carefully examining the initial DEX for loader patterns and then leveraging dynamic instrumentation tools like Frida, we can effectively intercept and dump the decrypted application code. This methodology allows reverse engineers to overcome a significant hurdle in analyzing protected Android applications, paving the way for further security assessments or understanding proprietary implementations.