Class imbalance in medical datasets may lead to the generation of biased Machine Learning models. Several methods are used to balance datasets but they do not consider the majority class samples while oversampling. Therefore, in this study, we proposed a novel technique called Borderline Counterfactual Oversampling with Noise elimination and Density Scoring (BCONDS). The method utilises isolation forest to remove the noisy samples from the majority class. Gower distance is used to find borderline minority class instances and extract their corresponding majority class neighbours. These neighbouring samples are then used to generate counterfactuals in order to enhance the separability of classes. The empirical analysis of four benchmark medical datasets indicates that our proposed technique outperforms other state-of-the-art techniques. On average, an improvement of 9.6% and 5.9% is recorded in the AUC and Gmean values of BCONDS when compared with other methods.