Multi-modal brain network (e.g., structural and functional connectivity networks) analysis utilizes complementary information of neuroimaging data with multiple modalities, thus help to gain a more comprehensive understanding of the brain’s function and structure. However, it is still a challenge task to effectively fuse the representation of structural and functional connectivity networks, with considering the intrinsic complementary of multi-modal data and unique characteristics of each modality data. To solve this problem, in this paper, we propose a multi-modal brain connectivity network (called M2BCN) learning framework that integrates the structural and functional connectivity networks for brain disease classification. Different with existing methods, we use structural connectivity from diffusion tensor imaging to induce more representative features of functional connectivity, using graph attention network method. Then we develop a novel deep learning framework to fuse features of two modality data for classification of brain diseases. The experiment results on an real epilepsy dataset demonstrate the effectiveness of the proposed M2BCN method.