Vim-F: Visual State Space Model Benefiting from Learning in the Frequency Domain

In recent years, State Space Models (SSMs) with efficient hardware-aware designs, known as Mamba deep learning models, have made significant progress in modeling long sequences. Compared to Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), Vision Mamba (ViM) methods have not yet achieved fully competitive performance. To enable SSMs to process image data, ViMs typically flatten 2D images into 1D sequences, inevitably ignoring some 2D local dependencies, thereby weakening the model’s ability to interpret spatial relationships from a global perspective. We believe that the introduction of frequency domain information can enable ViM to achieve a better global receptive field during the scanning process. We propose a novel model called Vim-F, which employs pure Mamba encoders and scans in both the frequency and spatial domains. Moreover, considering that Mamba remains essentially a Recurrent Neural Network (RNN), we question the necessity of position embedding in ViM and remove it accordingly in Vim-F. Vim-F has good scalability. As far as we know, its variant Vim-F(CF) is the first ViM model to use a convolution-free ViM encoder. Another variant, Vim-F(H), introduces a linear attention mechanism. This reduces the model’s sensitivity to the input sequence and achieves better performance.