This study investigates the efficacy of three feature extraction techniques: Discrete Cosine Transform (DCT), Local Binary Pattern (LBP), and Histogram Oriented Gradient (HOG) for malware detection. Utilizing the Malimg dataset, which comprises 2500 malware images across 25 families, this research aims to evaluate the effectiveness of different feature extraction techniques for malware classification through image processing methods. The methodology involves pre-processing the dataset, applying the three feature extraction techniques, and employing Principal Component Analysis (PCA) for dimensionality reduction. The extracted features are then evaluated using Euclidean distance to determine their accuracy in identifying malware. Results indicate that LBP consistently outperforms HOG and DCT in terms of accuracy and robustness across various training and testing splits. The study concludes with recommendations for future work, including hybrid feature extraction approaches and the integration of deep learning techniques to further improve malware detection systems. This research contributes to the field by providing a comparative analysis of feature extraction methods, highlighting the potential of LBP for reliable malware detection.