The application of deep learning in classifying malicious network traffic is prevalent. However, our paper highlights two frequently neglected issues in current methodologies: first, network traffic data need to be truncated and zero-filled to adapt to the training process of deep learning, which brings a certain degree of information loss; second, individual packets of network traffic will show different importance in the classification task due to their own data quality and location. To solve these two problems, a multimodal attention network is proposed to mitigate the information loss by fusing data from two different modalities in the network traffic and solving the imbalance of data from different modalities in the training process using a regularization method. Additionally, we incorporate a dual attention module to assess the significance of each packet, enabling the model to prioritize and emphasize the most crucial ones. To evaluate the model’s effectiveness, we test it on three openly accessible datasets: UNSW-NB15, ISCXIDS2012, and CIC-DoS2017. Our findings reveal that the model presented herein surpasses several conventional deep learning approaches in terms of classification outcomes.