P-Vit: A Simplified Vision Transformer Model Based on FFN and Simple Attention

Vit (Vision Transformer) has proved the advantages of its effectiveness and versatility in the existing image recognition field. However, the Vit model is still more complicated in the feed-forward neural network module, i.e., the FFN module, and the encoder module of the Transformer. Although the original GELU activation function in the FFN module has fast convergence speed and high accuracy, the GELU activation function involves exponential arithmetic, which has a large amount of computation. And the adjustment of the parameters is more difficult, which is also prone to cause a series of problems, such as over-fitting. Secondly, the encoder layer of the Transformer model in the Vit module is more complex, which makes the processing of input data more difficult. The image classification method P-Vit proposed in this paper, which incorporates linear activation function, solves the above problems by adding a simple attention module, which applies the PRELU activation function to the model. The PRELU used in this paper has stronger linear expression ability, can adapt to different data distributions and task requirements, and can solve the common dead neuron problem of activation functions. Besides, in this paper, we ported the Vit model to an Intel Arc graphics card for the first time for experiments and compared it with Nvidia related graphics cards.