Purpose: This study evaluated the MedSAM segmentation model combined with pre-trained models and machine learning for segmenting, extracting features, and classifying lesions in mammograms. The main objective was to evaluate MedSAM’s inference ability in the context of breast cancer detection, thereby lowering the amount of annotation work needed by radiologists. The goal of this method is to increase the effectiveness and precision of segmenting and categorizing breast lesions according to the Breast Imaging Reporting and Data System (BI-RADS). Methods: VinDr Mammo public mammogram dataset with 125 images categorized into BI-RADS lesions 3, 4, and 5, and density A and B was used. Images were pre-processed and segmented using MedSAM, with bounding box information as the prompt. Features were extracted from the segmented output using pre-trained models and applied to machine learning classifiers, including Support Vector Machine, Logistic Regression, and Random Forest models. Bayesian optimization was employed to enhance classification. Results: Majority of the models performed well using this novel approach with a limited dataset. The combination of the ResNext_50 model and the SVM model yielded the best results with an Accuracy of 96.15%, Precision of 96.58%, Sensitivity of 96.15%, and Specificity of 96%. Despite the small dataset size, results were encouraging, demonstrating MedSAM’s effectiveness as an inference model. Conclusion: MedSAM is a robust inference model, even when trained on a smaller mammogram dataset. Integrating MedSAM with machine learning and pre-trained models promises efficient lesion segmentation and classification, significantly reducing annotation costs and time, and offering a practical solution for medical image analysis with limited annotated data.