The emerging field of audio classification within encrypted domains responds to the growing need for data analysis techniques that protect privacy in industries handling sensitive audio information, like secure communications and healthcare. While image classification has benefited from CNNs’ strong pattern recognition capabilities, traditional audio classification has mainly relied on time-domain signal processing techniques. However, the development of this novel approach was motivated because the existing methods frequently fail to preserve data privacy during analysis. This paper presents a novel approach for audio classification which uses spectrograms and CNNs. Image-based CNNs can be applied to audio data using this technique, which turns audio signals into MEL spectrogram images. Sensitive data privacy is maintained throughout the process by performing neural network operations using the CryptoDL framework. The Python library TenSEAL implements homomorphic encryption, a crucial element of this methodology that enables secure computations on encrypted MEL spectrograms. This approach shows promise in maintaining the privacy of audio data. The results show high accuracy, especially for datasets such as TESS and RAVDESS. The benefit of handling encrypted data shows its potential for applications with severe data privacy policies.