With the rapid development of intelligent education, Computerized Adaptive Testing (CAT) has become a key tool for assessing student abilities. By dynamically adjusting exercises, CAT can accurately measure student abilities with fewer exercises, enabling personalized assessment and improving efficiency. However, traditional CAT methods mainly rely on statistical exercise properties, lacking deep semantic understanding and contextual reasoning, which limits personalization, interpretability, and adaptability. As a result, both educators and students find it difficult to trust the system’s selection process. To address this, we propose a novel CAT exercise selection method based on Large Language Models (LLMs), termed LLMCAT. LLMCAT leverages LLM’s semantic reasoning abilities to better understand and select exercises. The framework uses a two-step process: a traditional CAT algorithm firstly generates a pool of candidate exercises, and then a fine-tuned LLM ranks the exercises to select the most appropriate ones. This approach ensures both personalized selection and algorithmic stability. In addition, LLMCAT incorporates Low-Rank Adaptation (LoRA) to create an efficient fine-tuning pipeline, enabling quick adaptation of the LLM to the CAT task while reducing computational costs. Experimental results demonstrate that the proposed LLMCAT enhances the transparency and trustworthiness of the exercise selection process, offering new insights for the future development of intelligent educational assessment systems.