This paper explores the paradox of Artificial Intelligence (AI), particularly Machine Learning (ML), in modern medicine, promising enhanced diagnostics and treatment while in many instances also leading to a profound loss of transparency and interpretability. Complex AI systems such as Automated Decision-Making Systems (ADMS) increasingly guide critical clinical decisions; however, their opaque “black box” nature challenges traditional medical ethics centred on transparency, due process, accountability, and patient autonomy. ADMs often make recommendations that cannot be fully understood or explained due to the system’s opacity. This raises both legal and ethical concerns. For example, the ability of clinicians to justify clinical decisions informed by AI when they themselves lack insight into its reasoning raises concerns about clinical legal liability. This, in turn, raises ethical concerns regarding patient autonomy if patients are unable to meaningfully make informed choices on treatments due to the opacity of decisions made by AI. The extent to which patients should consent to and place their trust in life-saving technologies whose inner workings remain opaque remains an open debate. These technologies, arguably, can diminish patients’ rights to knowledge, autonomy, and dignity. Such concerns underscore the fragility of trust within clinical relationships and expose the limitations of prevailing frameworks for medical explanation (MedX) and informed consent. This paper examines the role of explainability in AI-driven medicine, arguing that while explanation is essential, it must be balanced against the clinical utility of high-performing, opaque systems. To address this tension, the paper proposes the Promethean Threshold Heuristic (PTH)—a normative framework designed to reconcile the need for explanation with clinical efficacy. Drawing on Promethean myth, it frames the ethical dilemma posed by AI as one of balancing risk, knowledge, and human dignity. Ultimately, it calls for principled AI governance in healthcare—one that preserves trust, prioritises transparency where feasible, and recognises the trade-offs inherent in deploying powerful but opaque technologies.