The evaluation of handwritten answers has traditionally relied on human judgment and rule-based systems, often leading to inconsistencies in scoring. This project proposes an advanced evaluation framework leveraging large language models to enhance accuracy and fairness. Our method introduces a two-source comparison approach by combining retrieval-augmented generation to generate model answers from a knowledge base and an answer key. This comparison enables a better understanding of the answer context. We also employ a two-level evaluation strategy involving large language model-based evaluation and semantic similarity techniques. This hybrid approach ensures a more comprehensive and nuanced assessment of handwritten answers, with the aim of reducing bias and improving grading consistency. Our evaluations show that the average semantic similarity of the student’s answer with answer key is approximately 10.63% higher than that of the generated model answers while the average score given by large language model remains the same. These metrics demonstrate that final evaluation that uses weighted averages can be adapted to teachers’ specific requirements according to the subject matter.