Cervical cancer remains a major global health burden, yet its development typically involves a long precancerous stage, during which effective screening and intervention can dramatically reduce mortality. Clinical practice relies on a combination of HPV testing, cytology, and colposcopy as complementary modalities for early detection. However, existing automated approaches often treat these modalities separately or reduce cytology to binary outcomes, limiting their clinical value. Moreover, heterogeneity in clinical reports poses challenges for reliable multimodal integration. We propose a two-stage multimodal deep learning framework with transfer learning for four-class cervical lesion classification, encompassing inflammation, low-risk lesion, high-risk lesion, and cancer. In the first stage, segmentation-informed pretraining is performed on colposcopy images with pixel-level annotations and HPV results, enabling the backbone to learn both structural and semantic representations. In the second stage, the pretrained backbone is fine-tuned for classification using multimodal fusion of colposcopy images, HPV reports, and cytology reports. To address variability in clinical text, large language models are employed for report normalization, improving consistency and facilitating reliable fusion. Experiments on two clinical datasets show that our method surpasses strong baselines. Ablation and visualization further confirm the benefits of multimodal fusion, LLM-based normalization, and segmentation-informed pretraining, highlighting its potential for reliable cervical cancer screening.