Large Language Models (LLMs) with their immense power and widespread applications are being more and more applied in almost any field, ranging from more general topics to more sophisticated topics, such as law and medicine. Applying LLMs in medicine provides numerous benefits for patients, practitioners, and researchers. LLMs organize and distill knowledge out of a considerable number of unorganized documents and records, enabling researchers and practitioners to shed light on previously unseen aspects of the problems under consideration, as well as provide personalized aids to patients in different ways. In this chapter, we are going to investigate the methods and techniques for transferring pre-trained successful models to make medical LLMs explainable; these techniques are referred to as ‘Transfer Learning’ collectively and are amongst the most interesting topics of active research in AI today. In this chapter, we first provide an overview of LLMs; then we briefly discuss the building blocks of the discussion, including explainable AI and transfer learning; finally, after a discussion on medical LLMs, we explore some research and methods for 'Transfer learning for explainable AI in clinical LLMs’. This book chapter may not suffice to treat the subject comprehensively, and we invite the interested reader to consult many valuable resources available.