Does ChatGPT Consistently Return Secure Cryptographic API Code?

Large language models, such as ChatGPT developed by OpenAI, have significantly enhanced productivity in various software development tasks. Despite these advancements, the security of code generated by those models remains a concern. This study investigates the security of ChatGPT-generated cryptographic API code snippets in four popular programming languages: Java, Python, C, and JavaScript. Our findings reveal that ChatGPT, on average, produces 1.38% insecure cryptographic code responses across these languages. Notably, Java exhibited the highest percentage of cryptographic API misuses, with 3.39% of the generated code responses containing security flaws. Despite its well-known security vulnerabilities, a recurring issue across all languages is the use of Electronic Codebook (ECB) mode for encryption. Symmetric encryption is the most problematic area, with a frequency of 2.82% of misuses identified. These results highlight the importance of rigorous evaluation and caution when leveraging ChatGPT for obtaining cryptographic API code examples.