Speech Emotion Recognition (SER) plays a pivotal role in enabling natural and responsive human-machine interactions for real-time applications. While state-of-the-art SER systems leverage large self-supervised learning models (e.g., wav2vec 2.0, HuBERT) to achieve high performance, their computational complexity and resource demands hinder deployment in latency-sensitive scenarios. To address this challenge, we propose Self-Layer-Wise-Distillation (SLWD), a novel framework for efficiently distilling knowledge from transformer-based teacher models into compact, high-speed student models. Our approach preserves the teacher’s representational power while drastically reducing computational overhead. Experiments demonstrate that the SLWD student model achieves competitive performance with the teacher—using only 36% of the parameters—and delivers a 1.5 \(\times \) inference speedup, making it ideal for edge deployment. This work bridges the gap between accuracy and efficiency in SER systems, advancing their practicality for real-world applications.