Current speaker verification systems often rely on neural networks like Gaussian mixture models or x-vector systems to extract speaker representations. While effective, these methods struggle to capture long-range audio dependencies and are not optimized for edge deployment due to computational inefficiency. In this paper, we propose AMMEX (Audio Mamba Mixture of Experts) - a lightweight and accurate architecture that combines state-space Mamba Encoders with a Mixture of Experts (MoE) framework. AMMEX eliminates self-attention, enabling real-time modeling of both local and global speech contexts with minimal resource requirements. By dynamically routing audio input to specialized expert encoders, the model enhances speaker embedding discriminability and inference efficiency. Evaluations on VoxCeleb1, Vivos, and CN-Celeb demonstrate that AMMEX significantly outperforms TDNN and Transformer baselines - offering state-of-the-art accuracy in a resource-efficient design suitable for on-device speaker verification in edge and IoT environments.