Multimodal emotion recognition is increasingly vital for affective computing and mental health applications. However, redundant features within and across modalities hinder effective fusion. Current research predominantly focuses on designing complex fusion architectures, with limited consideration given to the issue of feature redundancy. To address this limitation, we propose the Modality-Orthogonalized Fusion Architecture (MOFA), a novel approach that extends regularization to MERC to enhance overall performance. Specifically, we first employ modality-specific encoders to extract latent features from each modality and the modality-aligned gated adapter to align the features into unified feature dimension. Secondly, we propose an orthogonal representation strategy, which promotes intra-modality feature decorrelation and inter-modality semantic alignment. Finally, we propose a lightweight modality-guided fusion mechanism, where the video modality explicitly guides the inter-modal fusion process, enabling efficient and discriminative multi-modal feature integration. We evaluate MOFA on the EAV dataset, the first benchmark comprising electroencephalography (EEG), audio, and video modalities, facilitating MER enriched by neurophysiological signals. Extensive experiments demonstrate that the proposed method has superior performance compared to state-of-the-art methods, achieving an accuracy of 81.15% and an F1-score of 81.15%.