This paper presents a two-stage automatic scoring approach for handwritten answers in Japanese, English, and mathematics, targeting elementary and junior high school students. A dataset of handwritten answers from more than 480 students was collected on electronic paper devices and manually graded by teachers (human score). In addition, these answers were scored automatically in two stages: first through handwriting recognition and then through scoring, eliminating the need for teacher intervention. State-of-the-art deep neural network models, corresponding to each answer type, were used for the handwriting recognition stage. Subsequently, the proposed scoring models assign a score (machine score) for each answer based on recognition outcomes. The machine scores were then compared with the corresponding human scores to evaluate the performance. The experiment results show that, on average, approximately 80% of the answers were successfully scored automatically. Consequently, the average human scoring rates were approximately 15% for Japanese, 10% for English, and 17% for mathematics. This means that with automatic scoring less than 20% of the answers require human evaluation, compared to a scenario where all answers are scored manually. The analysis further reveals an average risky scoring rate of 2% (when the machine score is higher than the human score). While this rate should be further reduced, human scoring also exhibits a similar level of variance. These findings underscore the efficacy of the proposed automatic scoring approach in alleviating the teachers’ grading burden.