Large Language Models (LLMs) are extremely popular nowadays due to their power to recognize, summarize, translate, predict and generate text and other content. These capabilities have paved the way for the development of exceptional real-life applications. This paper attempts to present, using ChatGPT only as one example out of the many LLMs that currently exist or are under development, the several advantages that these applications can provide, but at the same time the concerns that their use or their development might raise in terms of cybercriminality. The central argument of this paper is that generative AI is not a magical solution. If deployed maliciously or without proper diligence, generative AI applications could cause unfathomable damage. Our analysis categorizes AI language models-related crime into two large groups—crimes with AI language models, and crimes against/on AI language models. These categories highlight serious flaws that can be associated with cybersecurity issues. Additionally, this paper aims to examine whether the EU’s proposed AI Act, which represents the first-ever legal framework on AI, adequately addresses crimes involving or targeting AI language models. Furthermore, this analysis aims to investigate whether the EU Directive on liability for defective products equates AI language models with other types of software, a position we contend is not appropriate.