Short answer scoring (SAS) that automatically scores learner’ answers based on given rubric. Because SAS uses different rubrics for different prompts, it is necessary to generate training data for each prompt, which is time-consuming and labor-intensive. We propose a human-in-the-loop framework that achieves SAS with a minimal amount of training data through active data creation and efficient training. We fine-tuned the Sentence BERT model with a small dataset by using SetFit (Lewis Tunstall, et al. (2022).) and then apply active learning (AL) to an unlabeled data pool. The grader scores the data selected by AL, creating new training data that are then used to further fine-tune the models. For the experiments, we used six prompts from a Japanese SAS dataset. Our experimental results demonstrate that the proposed method outperforms previous studies in most cases. We also examined the potential challenges when applying this framework in practical scenarios. This framework significantly reduces the graders’ time and effort by enabling automatic scoring with minimal training data.