Rumor spreaders use the rapid spread of multimodal content, such as visual-textual posts, to disseminate malicious rumors on social networks. While existing studies have concentrated on extracting shared information from various data modalities, including text, images, social graph features, and knowledge graphs, they often overlook the significance of modality-specific information. We propose a novel modality perception rumor detection network for multimodal rumor detection to address this issue. Our network can capture shared information and modality-specific data from text, images, and social graph features. Given the prevalent noise in multimodal data, we employ a supervised contrastive learning method to filter out this noise. This approach allows our model to identify the key characteristics of rumors and non-rumors, thereby enhancing its generalization ability. Additionally, we have designed an adaptive method to adjust the loss weights for multiple tasks. This method allows auxiliary tasks to maximize the learning enhancement of the primary task without disrupting its training. Our experiments on two public datasets show that our proposed model effectively detects rumors and outperforms existing methods.