This work focuses on the selection of representative instances for the training set in malware detection. Opposed to random instance selection, the goal of instance selection algorithms is to remove noise and redundancy while preserving relevant data for solving the task. Experiments were conducted on two publicly available datasets containing metadata of Windows PE files, namely the EMBER and SOREL-20M datasets. The theoretical part describes data preprocessing methods, instance selection algorithms, and classification algorithms used in the practical part of this work. The practical part outlines the process of preprocessing datasets and main experiments related to the comparison of state-of-the-art instance selection algorithms. As part of the work, modifications to the parallel instance selection algorithm PIF were proposed and implemented, and these were also experimentally evaluated and compared with the results of state-of-the-art instance selection algorithms. Some of the modified versions ranked among the best in terms of reduction level as well as the ratio between accuracy and the size of the reduced sets. The best among the modified versions was the RPIF-AllKNN algorithm, which reduced the entire training set of the SOREL-20M dataset to 6.24% of its original size with an accuracy loss of 2.1%. The ratio between accuracy and the size of the reduced set was 14.43 and in terms of this metric, RPIF-AllKNN was the best among the compared algorithms.