<p>With the development of deep learning, there are two main approaches for speaker recognition: the two-stage framework, which uses neural networks to extract embedding features, and the end-to-end framework, which uses neural networks for classification. However, the two-stage framework is complex, and the end-to-end framework can be affected by inconsistent audio durations. This paper focuses on emotional speaker recognition, a challenging task due to the emotional variability in speech, and proposes an end-to-end framework based on multi-network stacking (MNS). The MNS method adopts a frame-level input approach, enabling it to effectively capture dynamic emotional variations while addressing the duration inconsistency problem and simplifying overall complexity. Its dual-branch architecture (DNN and CNN) processes both temporal and spectral features from frames. By concatenating these features, the model forms a comprehensive representation that enhances speaker recognition even in emotionally variable speech. It also enhances information extraction for long-term audio by calculating the importance of each frame feature. Experimental results on both the Mandarin Affective Speech Corpus (MASC) and the Interactive Emotional Dyadic Motion Capture (IEMOCAP) database demonstrate the effectiveness of MNS. It achieves 97.28% accuracy and 1.27% EER on the MASC, and 93.79% accuracy and 3.09% EER on the IEMOCAP, consistently outperforming the strong baseline ECAPA-TDNN.</p>