Since OpenAI’s 2016 Request for Research on Image to LaTeX, printed mathematical expression recognition has advanced significantly, with newer techniques like Vision Transformers further enhancing accuracy. However, evaluations often remain limited to a small set of datasets and primarily focus on LaTeX string similarity, neglecting semantic and visual aspects. To perform a broader evaluation, we benchmarked five models representative of the evolution of Vision Transformers across eight datasets over multiple modalities. Our proposed multimodal evaluation pipeline, PiE-MER, converts predicted LaTeX into MathML, Label Graphs, and normalized images. This range of representations allows us to compute additional metrics that better capture models’ syntactic, semantic, and visual accuracy. Correlation analysis revealed that BLEU and Levenshtein alone are insufficient for complete evaluations. We propose complementary metrics, including image-based Levenshtein, MathML-based scores, and graph evaluation. The MathNet model achieved the highest average performance across the datasets, with a 87% LaTeX edit score, thanks to its Convolutional Vision Transformers and multi-font training dataset. Our pipeline and benchmark can offer a clearer and more robust evaluation of future recognition systems. A GitHub repository containing the proposed evaluation pipeline is available at https://github.com/fwieckowiak/PiE-MER , enabling future authors to evaluate their own models and datasets.