Multimodal fusion offers a complementary perspective for understanding brain function and structure. Nevertheless, most existing multimodal fusion methods are unsupervised and thus ignore the diagnostic information. Previous supervised fusion models stack subjects from different diagnostic groups into a single feature matrix, resulting in potential interference between diagnostic groups. In this study, we propose a group-specific multimodal fusion method, called Group-specific Multiset Canonical Correlation Analysis with Logistic Regression (GMCCALR) that can effectively leverage group specific information by incorporating a logistic regression term into mCCA and utilizing a group-specific fusion strategy. Results in human neuroimaging data show that GMCCALR provide higher diagnostic accuracy across several psychiatric disorders, including schizophrenia, schizoaffective disorder, bipolar disorder and autism spectrum disorder. We also show the identified disorder-specific multimodal patterns are replicable across multiple cohorts. Overall, results demonstrate the reliability and generalizability of the proposed GMCCALR approach and its efficacy for identifying multimodal co-varying diagnostic patterns for psychiatric disorders.