Large labeled datasets and computational resources are frequently needed to fine-tune large language models (LLMs) for domain-specific tasks like legal reasoning, healthcare, and education. We suggest CReST-PA (Contrastive-Regularized Self-Training with Prototype Alignment), a novel framework that accomplishes effective domain adaptation with little supervision, as a solution to this constraint. By combining contrastive learning, semantic prototype alignment, pseudo-labeling, and self-training, CReST-PA promotes robust representation learning while reducing the possibility of confirmation bias that comes with self-training. To fine-tune a foundational LLM (such as Mistral-7B or LLaMA2-7B) using parameter-efficient techniques like Low-Rank Adaptation (LoRA), the framework starts with a small, labeled dataset. Then, by producing high-confidence pseudo-labels for unlabeled domain-specific data (such as transcripts, university question papers, and grading rubrics), it iteratively broadens supervision. We build class-wise prototype embeddings that serve as anchors in the latent space, promoting representations of similar classes to cluster around their respective centroids in order to improve the semantic understanding of the model. The learning process is guided by a dual loss function that combines prototype-aware contrastive loss and cross-entropy. In comparison to standard self-training and contrastive learning baselines, our experiments on university-specific educational datasets show that CReST-PA achieves significant performance gains in accuracy, F1-score, and label efficiency. Additionally, curriculum-aware iteration minimizes early-stage noise from pseudo-labels and guarantees progressive refinement. A promising path for low-resource domain adaptation in NLP is established by the results, which confirm that CReST-PA is a scalable, generalizable, and computationally efficient solution for fine-tuning LLMs with little annotated data.