SAFFNet: self-attention based on Fourier frequency domain filter network for visual question answering
摘要
The focus point of visual question answering (VQA) tasks is how to reasonably and effectively model the relationship between complex information like images. The mainstream methods primarily employ a Transformer-based self-attention mechanism. However, this self-attention mechanism is insufficient to screen and filter effective and ineffective features. To address this issue, we introduce two learnable filters, the adaptive and the text-modulated filters, for effective image feature filtering. Subsequently, the effects of two filters are validated in the spatial domain. Furthermore, to better distinguish differences among features, the filters are optimized and propose a self-attention based on Fourier frequency domain filter network (SAFFNet). SAFFNet first transforms image features into frequency domain by fast Fourier transform, obtaining global frequency features while improving the differences between features. Secondly, the above two filters are used to filter in the frequency domain to achieve the goal of global filtering. We also designed a filtering network combining the frequency and spatial domain to explore its role. Finally, this paper uses the grid and regional visual features extracted from regional proposals and carries out experiments. The extensive experimental results on VQA v2 and GQA demonstrate the SAFFNet model’s outstanding performance. The highest accuracy of the model is 71.75