Off-Line Handwritten Math Formula Layout Analysis by ConvGNN

This work aims to construct an end-to-end system that can accurately analyze the layout of a mathematical formula. To accomplish this, we proposed a deep-learning architecture that makes use of a Convolutional Graph Neural Network (ConvGNN)—a variant of a convolutional neural network that operates directly on graphs to label symbols and their spatial relationships within the context of mathematical formulas, while also factoring in the features of the neighbors and their interrelationships. Testing different models of ConvGNN and comparing with the classical method: a Multi Layers Perceptron (MLP), we confirm the effectiveness of ConvGNN, especially the spectral-based one, as a suitable way to model formula structure that a graph can represent. We demonstrate that our system competes with some related works in several experiments on CROHME, a handwritten math formula standard database. These tests revealed an accuracy of 82% for MLP, 23% for simple ConvGNN, and 92% for spectral-based ConvGNN.