Can VLM Understand Children’s Handwriting? An Analysis on Handwritten Mathematical Equation Recognition

Handwriting Mathematical Expression Recognition has several applications, including the potential to make Intelligent Tutoring Systems (ITS) more accessible to underserved regions. However, young children’s handwriting pose several challenges, even for instructors, calling for advanced approaches to understand their writings. This paper explores the potential of pre-trained Vision-Language Models (VLM) for recognizing numbers or mathematical expressions handwritten by children by comparing GPT-4V, LLaVA 1.5, and CogVLM on a dataset of 251 images. Results indicate that while pre-trained models offer promise, their performance without fine-tuning or zero-shot learning remains inadequate. Results reveal the challenges of utilizing pre-trained VLMs for recognizing children’s handwriting, particularly in educational settings. Issues such as poor handwriting and partial erasures pose difficulties for existing models, with GPT-4V’s safety system limitations hindering its efficacy. In general, CogVLM presented the best performance. GPT-4V exhibited superior performance in recognizing equations, but still struggles with handwritten content, highlighting the need for model refinement and data policies. This study contributes insights into the potential of existing pre-trained models in educational contexts and demonstrates the importance of, for example, fine-tuning to domain-specific datasets. Continued research is necessary to enhance VLM capabilities for educational support, particularly in children’s handwriting recognition.