Ensemble classifiers outperform single classifiers in terms of classification accuracy, however, constructing an ensemble classifier that is optimized for consistently achieving high accuracy presents numerous challenges. One significant challenge is determining the optimal size of the input space, which includes features and clusters, as well as selecting the suitable number and types of base classifiers that contribute to a high-performing ensemble. In this paper, we introduce a novel approach to tackle this problem effectively. Our approach implements feature selection through the application of feature weights. Features are assigned relative weights, and those below a predetermined threshold are pruned, leading to a reduced dataset. Clustering is then applied to group similar data points in a new subspace. We evaluate the similarity of generated clusters, removing any exceeding a specified threshold. This enables us to identify distinct and diverse clusters while eliminating redundant or highly similar ones. We train diverse base classifiers on the remaining data clusters and add them to the base classifier pool. We optimize the base classifier pool to generate an ensemble classifier with improved classification accuracy. We conducted experiments on 21 datasets which demonstrate the efficacy of the approach. We also recognize that explainable AI (XAI) can further improve our approach by providing transparency within the ensemble model. It can explain the role of each feature and cluster in influencing the model’s predictions. We discuss how XAI can aid in understanding the influence of each classifier.