Introduction

Modern Android applications frequently employ custom cryptographic implementations to protect sensitive data, often to obscure their methods from reverse engineers and attackers. While static analysis can reveal some clues, dynamic analysis with tools like Frida offers an unparalleled advantage in observing these functions during runtime. This guide provides an expert-level walkthrough on leveraging Frida to trace, understand, and ultimately bypass custom Android cryptography, covering both Java and native (JNI) implementations.

Setting Up Your Environment

Before diving into crypto analysis, ensure you have a proper Frida environment configured. This requires a rooted Android device or emulator, the Frida server running on it, and Frida tools on your host machine.

Prerequisites:

• Rooted Android Device/Emulator (e.g., AVD, Genymotion, or physical device).
• ADB (Android Debug Bridge) installed and configured on your host.
• Python 3 and pip installed on your host.
• Frida-tools installed via pip:

pip install frida-tools

Frida Server Setup:

Download the appropriate Frida server binary for your Android device’s architecture (e.g., frida-server-*-android-arm64 from Frida’s GitHub releases). Push it to the device, set permissions, and run it.

adb push frida-server-*-android-arm64 /data/local/tmp/frida-serveradb shell "chmod 755 /data/local/tmp/frida-server"adb shell "/data/local/tmp/frida-server &"

Understanding Android Cryptography Landscape

Android applications typically use the Java Cryptography Architecture (JCA) and Java Cryptography Extension (JCE) for standard cryptographic operations. However, many developers opt for custom implementations for several reasons:

• Security through Obscurity: Believing custom algorithms are harder to break (often a misconception).
• Performance: Implementing critical crypto in native code for speed.
• Bypassing Export Restrictions: Historically, some regions had restrictions on strong crypto, leading to custom solutions.
• Integration with Obfuscation: Embedding crypto logic within complex, obfuscated code paths.

Our goal is to identify these custom functions, regardless of whether they reside in Java or native libraries, and hook them to observe their inputs and outputs.

Frida for Java Crypto Tracing – The Basics

When an application uses standard Java crypto classes like javax.crypto.Cipher or java.security.MessageDigest, Frida can directly hook these methods to inspect parameters. This is often a good starting point to confirm if standard APIs are used.

Example: Hooking Cipher.doFinal

To see what’s being encrypted or decrypted by a standard AES cipher, you can hook doFinal.

Java.perform(function () {    var Cipher = Java.use('javax.crypto.Cipher');    Cipher.doFinal.overload('[B').implementation = function (input) {        console.log('Cipher.doFinal called with input (bytes):', Array.from(input));        var result = this.doFinal(input);        console.log('Cipher.doFinal returned (bytes):', Array.from(result));        return result;    };    Cipher.doFinal.overload('[B', 'int', 'int').implementation = function (input, inputOffset, inputLen) {        console.log('Cipher.doFinal called with input (bytes), offset, len:', Array.from(input), inputOffset, inputLen);        var result = this.doFinal(input, inputOffset, inputLen);        console.log('Cipher.doFinal returned (bytes):', Array.from(result));        return result;    };});

Save this as crypto_trace.js and run with:

frida -U -f com.example.app -l crypto_trace.js --no-pause

Replace com.example.app with the target package name.

Diving Deeper: Custom Java Crypto Implementations

The real challenge begins when applications implement their own encryption logic in custom Java classes. Identifying these often requires static analysis (decompilation with tools like Jadx or GHIDRA) to locate suspicious class names (e.g., CryptoUtils, EncryptionManager, ObfSecurity) or methods (e.g., encryptData, decryptMessage).

Example: Hooking a Custom encrypt Method

Suppose static analysis reveals a class com.example.myapp.CryptoUtil with a method public byte[] encrypt(byte[] data, byte[] key, byte[] iv).

Java.perform(function () {    var CryptoUtil = Java.use('com.example.myapp.CryptoUtil');    CryptoUtil.encrypt.overload('[B', '[B', '[B').implementation = function (data, key, iv) {        console.log('--- Custom CryptoUtil.encrypt called ---');        console.log('Plaintext (bytes):', Array.from(data));        console.log('Key (bytes):', Array.from(key));        console.log('IV (bytes):', Array.from(iv));        var result = this.encrypt(data, key, iv);        console.log('Ciphertext (bytes):', Array.from(result));        return result;    };});

This hook will capture the plaintext, key, and IV used by the custom encryption method, providing critical insights into the algorithm’s parameters.

Native Crypto Analysis with Frida (JNI)

Many performance-critical or security-sensitive cryptographic operations are moved into native libraries (.so files) to prevent easy reverse engineering via decompilers. Analyzing these requires a deeper understanding of native code and memory management.

Identifying Native Crypto Functions:

1. Static Analysis: Use tools like IDA Pro or GHIDRA to open the target .so library (e.g., libcustomcrypto.so). Look for functions related to JNI (e.g., Java_com_example_myapp_NativeCrypto_encrypt) or functions commonly associated with crypto libraries (e.g., AES_set_encrypt_key, EVP_EncryptInit_ex if OpenSSL is used).
2. String Search: Look for common crypto-related strings within the binary (e.g., "AES", "DES", "PKCS5Padding").
3. Exported Functions: Use nm -D libcustomcrypto.so on the library to list exported symbols, which might reveal JNI functions.

Hooking Native Functions with Interceptor.attach

Once you identify a native function’s address or symbol name, you can use Frida’s Interceptor.attach to hook it. Reading arguments requires understanding calling conventions (e.g., ARM64 uses X0-X7 for first 8 arguments, then stack).

Example: Hooking a Native encrypt Function

Assume we’ve found a native function at a specific address (e.g., 0x12345 relative to the base of libcustomcrypto.so) or by its symbol name custom_aes_encrypt.

Interceptor.attach(Module.findExportByName('libcustomcrypto.so', 'custom_aes_encrypt'), {    onEnter: function (args) {        console.log('--- custom_aes_encrypt called ---');        // Assuming the function signature is similar to:        // int custom_aes_encrypt(char* plaintext, int plaintext_len, char* key, int key_len, char* output_buffer)        // On ARM64: arg0 = X0, arg1 = X1, etc.        this.plaintextPtr = args[0];        this.plaintextLen = args[1].toInt32();        this.keyPtr = args[2];        this.keyLen = args[3].toInt32();        this.outputBufferPtr = args[4];        console.log('Plaintext length:', this.plaintextLen);        console.log('Key length:', this.keyLen);        console.log('Plaintext:', this.plaintextPtr.readByteArray(this.plaintextLen));        console.log('Key:', this.keyPtr.readByteArray(this.keyLen));    },    onLeave: function (retval) {        // Assuming retval is the length of the ciphertext or a status code        console.log('Ciphertext length:', retval.toInt32());        // If output buffer is pre-allocated and written to by the function        if (this.outputBufferPtr && retval.toInt32() > 0) {            console.log('Ciphertext:', this.outputBufferPtr.readByteArray(retval.toInt32()));        }        console.log('--- custom_aes_encrypt finished ---');    }});

Important Considerations for Native Hooks:

• Calling Conventions: Be mindful of the target architecture (ARM, ARM64, x86) and its calling convention to correctly read arguments from registers or the stack.
• Memory Management: Arguments passed as pointers (`char*`, `byte*`) require readByteArray() or readUtf8String() to inspect their content.
• Return Values: The retval in onLeave can be cast to toInt32(), toPointer(), etc., depending on the expected return type.
• Function Overloading: Native functions typically don’t have direct "overloads" like Java, but different function names might exist for similar operations.

Automating and Refine Your Analysis

As you gather more information, you can refine your Frida scripts. For instance, if you discover an initialization vector (IV) is always used, you can add hooks to where the IV is generated or passed to the cipher. You can also write utility functions within your Frida script to convert byte arrays to hex strings for easier readability or even perform decryption if you manage to extract the key and algorithm.

Example: Bytes to Hex Utility

function bytesToHex(bytes) {    return Array.from(bytes, function(byte) {        return ('0' + (byte & 0xFF).toString(16)).slice(-2);    }).join('');}console.log('Key (hex):', bytesToHex(this.keyPtr.readByteArray(this.keyLen)));

Conclusion

Frida is an indispensable tool for dynamic analysis of Android applications, particularly when dealing with custom cryptographic implementations. By mastering its capabilities for both Java and native hooking, you can dissect obfuscated crypto logic, extract critical parameters like keys and IVs, and ultimately gain full control over how an application protects its data. This deep dive into an application’s runtime behavior is crucial for security researchers and penetration testers seeking to thoroughly evaluate the resilience of mobile applications.