Breast cancer is the most common cancer in women. It develops slowly and silently, making early detection crucial for effective treatment. Machine learning provides models for predicting breast cancer based on data from previously diagnosed patients. The collected data are not all equally important, some features contribute more effectively to predict breast cancer better, while others slow down the model and reduce its performance, highlighting the importance of feature selection. Our proposed approach adopts the Weighted Soft Voting classifier, using XGBoost, Random Forest, and Decision Tree classifiers for feature selection. First, hyperparameter optimization is performed to select the best hyperparameters for each classifier that will participate in the voting. Weights are then assigned to the three classifiers based on their individual performance. Next, the Weighted Soft Voting classifier is used to evaluate the importance of the features. Finally, a search for the optimal number of features to keep is performed, and a dataset with the selected features is obtained. Our feature selection approach enhances the performance of the Weighted Soft Voting classifier, as well as the individual performance of XGBoost and Decision Tree.