The success of the applications of so-called “artificial intelligence” (AI) are unmistakable. AI is used in numerous fields: entertainment, art, and in many technical systems, AI-supported assistance systems are in use. Despite all the remarkable achievements, however, it has not yet been possible to provide a safety case for an AI or even to obtain approval, for cases, where the AI is solely responsible for a safety-relevant decision. The discussion about AI-supported systems has also been raised in railway technology, especially in automated driving. In this paper we explain by a thought experiment for a particular problem from safety-related communication systems, whose solution is known, the advantages of model-based AI engineering. In the thought experiment we discard the underlying results from coding theory and treat the algorithm as a black box, like a complex AI based algorithm, and exploit only empirical and structural properties. So, we can compare such results with the mathematically correct results and explore the gap between theory and practice. We argue why such an approach may “close the gap” between AI performance and safety requirements.