Cervical cancer remains a significant global health concern, emphasizing the need for effective diagnostic methods. Despite advancements in Vision Language Models, challenges persist in incorporating cytological knowledge, ensuring data relevance, and maintaining accuracy when aggregating visual information. Current methods often struggle to handle fine-grained morphological details and the complex relationships between images and textual knowledge. In this paper, we present a novel framework for cervical cell classification that combines attribute descriptors with cytological knowledge for enhanced morphology recognition. Our approach leverages the Vision Large Language Model to generate descriptions for each cervical image and pretrain image and text encoders, improving both image understanding and cytological context. We introduce Attribute Descriptors Extraction using LLMs and Retrieval-Augmented Generation to generate detailed descriptors that capture important cytological features while minimizing irrelevant information. Additionally, we propose Optimal Attribute Descriptors Matching to dynamically align textual descriptors with image features, enhancing prediction accuracy, interpretability, and cytological relevance. Experimental results demonstrate the superior performance and generalizability of our method with varying amounts of labeled data. The code is publicly available at https://github.com/feimanman/CervicalCellClassifier