Scholarlink

<p>Recognizing handwritten mathematical expressions (HMER) is difficult because of challenges with the 2D structure of the expressions, inconsistent scales of the symbols, and complex spatial relationships of the symbols. In this work, self-supervised pretraining is used for representation and attention learning without requiring labeled data; however, the downstream Transformer-based recognition stage is trained on labeled expression–LaTeX pairs, thereby reducing reliance on large labeled datasets. We first pretrain an image encoder with a mixed global and local contrastive loss for the model to learn both a holistic representation and fine representation. An essential part of this paper is a new self-supervised attention reconstruction under progressive masking. The attention learning selects semantically meaningful regions of focus (operators, exponents, nested mathematical notation, etc.) without supervision. The progressive masking curriculum ultimately allows the network to learn to deal with missing or occluded visual information, which ultimately helps with structural understanding. The complete pipeline consists of (1) a self-supervised encoder pretraining, (2) self-supervised attention reconstruction under progressive masking, and (3) a supervised fine tuning of a transformer decoder in order to produce LaTeXsequences. To evaluate how effective the proposed progressive attention mechanism is for improving the HMER, extensive experiments are carried out on the CROHME benchmarks datasets. The proposed method achieves competitive performance compared to a broad set of recent state-of-the-art supervised and SSL baselines. It obtains expression recognition rates of 66.63%, 65.91%,66.97%, and 62.96% on the CROHME 2014, 2016, 2019, and 2023 test sets, respectively. Our codebase can be found <a href="https://github.com/pramaniksandip/MoCoMER-V2">here.</a></p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

MoCoMER-V2: mask & match for self-supervised attention to recognize handwritten math expression

  • 期刊论文

  • 发表日期：

    2026年7月27日

查看全文
  • Sandip Pramanik,
  • Shree Mitra,
  • Ritabrata Chakraborty,
  • Nibaran Das

摘要

Recognizing handwritten mathematical expressions (HMER) is difficult because of challenges with the 2D structure of the expressions, inconsistent scales of the symbols, and complex spatial relationships of the symbols. In this work, self-supervised pretraining is used for representation and attention learning without requiring labeled data; however, the downstream Transformer-based recognition stage is trained on labeled expression–LaTeX pairs, thereby reducing reliance on large labeled datasets. We first pretrain an image encoder with a mixed global and local contrastive loss for the model to learn both a holistic representation and fine representation. An essential part of this paper is a new self-supervised attention reconstruction under progressive masking. The attention learning selects semantically meaningful regions of focus (operators, exponents, nested mathematical notation, etc.) without supervision. The progressive masking curriculum ultimately allows the network to learn to deal with missing or occluded visual information, which ultimately helps with structural understanding. The complete pipeline consists of (1) a self-supervised encoder pretraining, (2) self-supervised attention reconstruction under progressive masking, and (3) a supervised fine tuning of a transformer decoder in order to produce LaTeXsequences. To evaluate how effective the proposed progressive attention mechanism is for improving the HMER, extensive experiments are carried out on the CROHME benchmarks datasets. The proposed method achieves competitive performance compared to a broad set of recent state-of-the-art supervised and SSL baselines. It obtains expression recognition rates of 66.63%, 65.91%,66.97%, and 62.96% on the CROHME 2014, 2016, 2019, and 2023 test sets, respectively. Our codebase can be found here.