Pre-trained language models (PLMs) have revolutionized natural language processing (NLP) and are now being used in recommendation systems. PLMs learn universal representations from large corpora and can be fine-tuned for specific recommendation tasks with minimal data. This paper presents a taxonomy of PLM-based recommender systems, categorizing them by training paradigms and data types. We also discuss different LM learning goals, including pre-training, language generation, and sequence classification, and how they can be applied to recommendation systems. The effectiveness of these paradigms has been shown in various recommendation tasks, but there are still problems to be addressed, such as language bias and fact consistency in language generation tasks, transfer and injection of domain knowledge, and maintaining and updating large-scale models without sacrificing accuracy or efficiency. The paper also discusses open research questions, such as multi-modal recommendations and privacy concerns. This survey provides a structured way to analyze the effectiveness of different approaches to using PLMs for recommendation systems and provides a valuable resource for researchers and practitioners looking to improve their own recommendation systems.