Diagnosing COVID-19 from chest X-ray images manually has been a challenging task even for expert radiologists. The main challenge for radiologists during the recent pandemic is to distinguish between COVID-19 and other lung diseases such as Pneumonia. The challenge is due to overlapping symptoms in COVID-19 and Pneumonia chest X-ray images such as Medical Ground-Glass Opacities, Consolidation, Bilateral Infiltrates, Reticular Patterns, Pleural Effusions, and Peripheral Distribution. Machine learning can be applied to build automated diagnostic learning models for distinguishing between COVID-19 and Pneumonia. However, overlapping symptoms in Chest X-ray images throws challenge to machine learning classifiers to distinguish between COVID-19 and Pneumonia. The quality of any machine learning model is highly dependent on the quality of features which are obtained using feature selection or feature extraction methods. To build an efficient machine learning model, the features which are used to build the model have to be representative of the lung diseases considered. Thus, feature engineering plays a very crucial role in building a good machine learning model. In this paper, we propose a new feature extraction method for extracting representative features from medical images which can substantially improve the performance of machine learning classifiers. For experimental analysis of the proposed method, we have used the most recent benchmark dataset, i.e. Novel COVID-19 Chest X-ray dataset which is available publicly at Kaggle consisting Normal, Pneumonia and Covid-19 chest X-ray images. Experimental analysis proved that the performance of machine learning classifiers using the proposed feature extraction method outperformed DCNN based feature extraction using VGG-16, Inception V3 and Exception models when multiclass classification is performed. The performance of MLP classifier is studied using the features extracted from VGG-16, Inception V3, Exception DCNN models. It is proved that the SVM classifier performance using feature extracted with the proposed feature extraction method is better than those obtained using DCNN models.