Human emotion recognition (HER) has gained considerable attention in recent years, driven by advances in machine learning, acoustic signal analysis and natural language processing. The advancements in HER using speech as a principal modality is addressed in this survey systematically. The importance of emotional intelligence in HCI, social robotics and mental health assessment in this review comprehends a complete analysis of approaches including feature extraction techniques, classification algorithms and data representation used in the field of speech emotion recognition. Furthermore, in this survey, existing methods into traditional rule-based systems, machine learning algorithms and state-of-the-art deep learning frameworks, stressing on the strengths and limitations are discussed. Additionally, thoughtful challenges such as the density of human emotions, the influence of contextual factors and the need of annotated datasets to train robust emotion recognition systems find their involvement in this work. Current trends in multimodal emotion recognition (MER) and the incorporation of speech with other modalities are also discussed to provide a complete view. This amalgamation of existing literature aims to notify future research directions in emotion recognition systems, enhancing their pertinency across varied fields.