Handwritten mathematical expression generation (HMEG) aims to synthesize handwritten samples with specific mathematical content, which has a wide range of applications and thus has important research value. However, current studies on HMEG are confronted with two bottlenecks: 1) dominant models are constructed in adversarial learning, which tends to suffer from the error propagation of recognition. 2) these models require diverse inputs, increasing both cost and complexity in practical applications. In this work, we propose a novel Spatial-Aware Feature Refinement Diffusion (SFRD) combined with a content prototype representation to address the challenging HMEG task. Specifically, we employ two structurally shared encoders to extract handwritten patterns and content features. The extracted patterns are then processed through the designed Spatially-Aware Feature Refinement Module (SAFRM), which refines the understanding of spatial information in the handwritten style using spatial attention and feature refinement. In SFRD, cross-attention is applied to capture the entanglement between the two feature types, guiding the diffusion generation process. Extensive experiments and in-depth analyses on widely used benchmark datasets clearly demonstrate the effectiveness of our proposed model. Our code is publicly available at:  https://github.com/Fyzjym/SFRD_Part .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

SFRD: Handwritten Mathematical Expressions Generation by Spatial-Aware Feature Refinement Diffusion

  • Yiming Wang,
  • Hongxi Wei,
  • Heng Wang,
  • Shiwen Sun

摘要

Handwritten mathematical expression generation (HMEG) aims to synthesize handwritten samples with specific mathematical content, which has a wide range of applications and thus has important research value. However, current studies on HMEG are confronted with two bottlenecks: 1) dominant models are constructed in adversarial learning, which tends to suffer from the error propagation of recognition. 2) these models require diverse inputs, increasing both cost and complexity in practical applications. In this work, we propose a novel Spatial-Aware Feature Refinement Diffusion (SFRD) combined with a content prototype representation to address the challenging HMEG task. Specifically, we employ two structurally shared encoders to extract handwritten patterns and content features. The extracted patterns are then processed through the designed Spatially-Aware Feature Refinement Module (SAFRM), which refines the understanding of spatial information in the handwritten style using spatial attention and feature refinement. In SFRD, cross-attention is applied to capture the entanglement between the two feature types, guiding the diffusion generation process. Extensive experiments and in-depth analyses on widely used benchmark datasets clearly demonstrate the effectiveness of our proposed model. Our code is publicly available at:  https://github.com/Fyzjym/SFRD_Part .