<p>Audio-visual emotion recognition, integrating information from various sources, enhances the comprehensive understanding and analysis of human emotions. However, existing methods in this field primarily focus on feature fusion, often overlooking the equally crucial stage of feature extraction. Typically, these methods employ pre-trained models to extract features during the initial phase, complicating modality-specific feature extraction and increasing system complexity and coordination challenges, which hinder effective audio-visual feature fusion. To address these issues, we propose an end-to-end audio-visual emotion recognition model, EAVFormer. For the audio modality, we input raw audio signals along with a mix of mel spectrograms and Mel Frequency Cepstral Coefficients (MFCCs), employing a hierarchical transformer technique that facilitates both shallow local and deep global feature extraction to fully harness complementary acoustic information, significantly enhancing the acoustic modality’s contribution. For the visual modality, we input raw video frames and employ a hierarchical transformer feature extraction method that integrates three-dimensional convolution and spatio-temporal self-attention mechanisms to address spatial and temporal redundancies and dependencies, thereby enhancing the expression of dynamic visual information. In the modal fusion stage, we construct dual-modality representations and encode inter-modal information while preserving modality-specific and meaningful intra-modal features. Testing on three popular datasets RAVDESS, CREMA-D, and CMU-MOSEI demonstrates that EAVFormer surpasses existing state-of-the-art methods in audio-visual emotion recognition accuracy, achieving rates of 95.17%, 86.13%, and 78.22%, respectively.</p>