Applying artificial intelligence (AI) in health brings about new changes in patients’ management and will bring efficiency in diagnosis, treatment, and other working processes. These improvements, however, give serious questions on patient’s privacy and protection of their data. Drawing from the literature, this paper aims to identify key ethical considerations in the use of AI in health care and the major conflict of interest that arise when considering the use of AI: the need to encourage innovation while also preserving patients’ rights to privacy. We look at how these particular AI technologies work, the issues of patient data privacy that come with the application of these technologies, and issues of ethics in use of data. The paper also covers aspects of privacy protection measures and proper ethical frameworks and guidelines to control privacy threats and facilitate technology development. In addition, it addresses the rationality of the current regulating processes and effectiveness of cooperation of all interested subjects in furthering the principles of ethical AI. Lastly, this paper seeks to give a brief look at dealing with AI in healthcare context, with the focus placed on the responsible and thoughtful approach to development.