LLMs promote digital health in a significant sense of the word because they produce more advanced medical text, interact with patients better, and interface with clinical issues in a supportive manner. Nonetheless, LLMs present significant legal, ethical and regulatory challenges over and above current regulations governing customary medical devices that might not be addressed. The current laws (General Data Protection Regulation of the European Union and AI Act of the United States) consider accountability, demonstrate transparency, and ensure patients are safe but are not compatible, descriptive, and surveying the post-market AI/ML systems in healthcare (Mello and Cohen in JAMA 333:1769–1770, 2025 [1]; Mennella et al. in Heliyon 10(4), 2024 [2]; Kolfschooten Med Law Rev 32:373–391, 2024 [3]). The chapter will evaluate the ways in which regulation and governance of LLMs in healthcare must be refocused in accordance with the principle of explainability, human control and dynamic compliance by considering the numerous interactions between the law and the AI systems that are in operation, which are dynamic and in a state of constant flux. We will dwell on the necessity to adapt regulation, share responsibility and openness of algorithmic choice, relying on the experience of Europe, America and Global South. In addition to the legal compliance, healthcare AI should be explainable, fair, and constantly evaluated, which is closely interconnected with the creation of ethical and socially acceptable technologies in healthcare.