Legacy systems continue to remain crucial for the business operations of many organizations around the world. However, for several reasons, there is an increasing pressure to modernize them. While the existing approaches to legacy system modernization have considerable drawbacks, the recent advancements in generative Artificial Intelligence (GenAI), and in particular Large Language Models (LLM), provide new opportunities. Therefore, this paper explores the capabilities of LLMs in converting procedural COBOL applications to a modern Java architecture, while maintaining the legacy system’s original functionality. An LLM-based framework for COBOL-to-Java conversion is developed and evaluated. The generated Java code was assessed using static code analysis and review by experienced developers. Results indicate, that LLMs can successfully capture the original business logic and convert COBOL applications to logically correct, clean and truly object-oriented Java code. However, the complete functional correctness of the generated Java code could not be verified in practice so far, and manual preparation of the input data as well as extensive prompt engineering were necessary to obtain these results.