In this paper, an end-to-end CNN architecture for music genre classification (MGC) using orthogonal feature fusion and orthogonal projection loss (OPL) will be proposed. First, MS-SSincResNet is used to learn several 2D representations and the corresponding embeddings with different time-frequency aspects. Local and global features are then extracted from these embeddings. The intra- and inter-feature orthogonal fusion (IIOF), which aims to enhance the diversity of the global feature and produce complementary information between local and global features, is used to fuse local and global features into a unified descriptor. Finally, a fully connected (FC) layer with softmax output and trained with OPL, which imposes orthogonality in the feature space to enforce inter-class separation as well as intra-class compactness through orthogonality constraints on the mini-batch level, is designed to improve the classification accuracy. The experimental results have shown that the proposed method outperforms other state-of-the-art methods on the ISMIR2004 music genre classification dataset.