Recognition accuracy of Automatic Speech Recognition (ASR) System primarily depends upon training and testing of underlying acoustic and language models. The corpus used for training and testing usually consists of speech recordings from different speakers, which are generally taken without considering their gender. Voice of males and females is quite different. Male voice has low pitch and high pitch period and for female voice it is vice versa. While collecting speech and data developing the ASR model these variabilities need to be considered else model may result into biased model for one group. This paper aims to evaluate and analyze the dissimilarity between voices of male and female speakers based on their acoustic features for an ASR system. Exclusive model for male and female voices have been developed and cross testing has been done. The results show that model developed for one gender is not efficient in recognizing speech from other gender. The comparison of gender-based recordings results in certain speech characteristics which need to be considered while training and testing ASR systems.