In speech emotion recognition (SER) literature, the common practice is to remove the silence frames before extracting the speech features. However, this work uses the RAVDESS database and investigates the relevance of silence as an emotion discriminating cue. We use an energy-based voice activity detector (VAD) to detect the silence and speech frame sequences. The silence frame sequences are then used to capture four different silence quantifying features: mean and standard deviation of silence duration, fraction of silence frames, and speaking rate. In this work, we perform two different experiments to extract the silence features: diversity-constrained and gross-level experiments. We then analyze the patterns of silence features for different emotion classes. The gross-level experiment is further validated with the different VAD thresholds. From the results, we observe that the extracted silence features show discriminating patterns across the different emotion classes. This finding emphasizes that silence captures crucial emotion recognition cues and should not be discarded in SER system development.