CryptoLLM: Harnessing the Power of LLMs to Detect Cryptographic API Misuse

We propose CryptoLLM, a novel static analysis tool leveraging large language models (LLMs) to detect cryptographic API misuse vulnerabilities. Integrating optimized code slicing with fine-tuned LLMs, CryptoLLM achieves superior detection capabilities. After evaluating four models, we recommend CodeT5. CryptoLLM outperforms existing rule-based tools such as CryptoGuard, CogniCrypt, and SpotBugs on the CryptoAPI-Bench dataset (F1 score: 0.935). For unseen real-world Android apps, with a 20-minute analysis limit, CryptoLLM achieved the highest F1 score of 0.898, analyzing all apps without errors, while other tools failed to analyze a significant proportion, with CryptoGuard’s highest F1 score at 0.645. Although CryptoLLM ’s performance initially dropped to 0.749 F1 score on mutated code, retraining with augmented data improved it to 0.988, demonstrating adaptability across diverse datasets.