<p>The rapid integration of artificial intelligence (AI) into clinical practice presents novel legal and ethical challenges that traditional legal frameworks—predicated on human autonomy and predictability—struggle to address. This paper examines the evolving legal liability of physicians and the necessity for new governance models in the era of medical AI. We argue that conventional, static regulatory approaches are insufficient for the fast-evolving nature of AI technologies. Instead, we advocate for “Agile Governance,” a dynamic framework involving continuous rule evaluation by multiple stakeholders, including government bodies, corporations, academic societies, and patient/public representatives. Through the analysis of hypothetical malpractice scenarios, we explore key legal issues: the physician’s role as the ultimate decision-maker, the duty of explaining AI usage, and the emerging responsibilities of AI vendors. Our analysis highlights that liability is increasingly dispersed among physicians, medical institutions, and vendors, complicating the attribution of responsibility. Drawing parallels with the autonomous driving sector, we propose a systemic approach to safety and liability. This includes establishing clear performance standards, creating robust accident investigation mechanisms to facilitate information sharing, and designing collective compensation systems for victim relief. We conclude that physicians must actively participate in designing these social systems to maximize the benefits of AI while appropriately managing the associated risks.</p>