Speech Emotion Recognition (SER) is essential for enriching user engagement in human-computer interaction, with applications in emotional computing, mental health monitoring, and intelligent customer service. However, many recent SER approaches rely on attention mechanisms, which, despite their excellent performance, incur high computational complexity, limiting real-time deployment. To address this, we propose MSNet, a novel hierarchical framework that leverages the natural structural characteristics of speech. MSNet processes emotional features progressively from coarse-grained to fine-grained levels, capturing nuanced emotional expressions across multiple stages. By replacing traditional attention mechanisms with a Mamba module, which employs a selective state-space model with linear-time complexity, MSNet significantly reduces computational overhead while maintaining robust performance. Extensive experiments on the publicly available IEMOCAP and MELD datasets demonstrate that MSNet achieves superior emotional classification accuracy with substantially lower computational costs compared to the baseline method, underscoring its potential as an efficient and scalable backbone for advanced speech processing applications.