A Multimodal Patent Summarization Model with Cross-Modal Attention and Dynamic Gating Fusion
摘要
Patents contain a large number of technical terms and drawings. Existing models struggles to capture the fine-grained semantic correlation between text and images during processing. This can easily result in the loss of technical details and make the model susceptible to invalid image information, affecting the accuracy and efficiency of summary generation. In this paper, we propose a dual-mechanism model (DMAF-MS), which is based on cross-modal attention and dynamic gating fusion. This model explicitly models the semantic alignment of text words and image regions through the cross-modal multi-attention mechanism. It also adaptively adjusts the fusion weights of graphical and textual features using the dynamic gating network. At the same time, the model introduces an image mask mechanism to optimize the efficiency of attention calculation. Experiments on the self-constructed MPSum-MM dataset demonstrate that the abstracts generated by our model significantly outperform the baseline approach in terms of technical term accuracy and modal integration. This verifies the model's ability to accurately depict complex graphical relationships and significantly improves the efficiency and accuracy of patent analysis.