Speaker Emotion Recognition (SER) is a critical aspect of human-computer interaction, enabling intelligent systems to perceive and respond to human emotions effectively. However, existing SER models struggle to capture subtle emotional cues, handle high-dimensional speech data, and maintain computational efficiency. To address these challenges, this study proposes the Quantum Classical Speaker Model (QCSM), a novel framework that integrates quantum computing with deep learning to enhance SER performance. Unlike conventional approaches, QCSM employs quantum circuits for advanced feature extraction, leveraging quantum parallelism to encode complex emotional patterns while utilizing classical neural networks for robust classification. This hybrid architecture is designed to optimize both computational efficiency and expressive power, overcoming the limitations of purely classical or quantum models. The proposed model is rigorously evaluated on multiple benchmark SER datasets and compared against state-of-the-art classical and quantum approaches. Extensive experiments demonstrate that QCSM significantly outperforms existing quantum-based SER models, achieving superior accuracy and generalization across diverse emotional expressions. By introducing a quantum-enhanced feature learning mechanism and an optimized hybrid architecture, this work bridges the gap between quantum computing and deep learning in SER, contributing to advancing emotion-aware AI systems and setting a new direction for future research in quantum-enhanced affective computing.