Background <p>Traditional teaching in undergraduate dermatology and venereology often prioritizes theoretical knowledge over clinical competency development. To address this limitation, we implemented a hybrid instructional model that combines scenario-based simulation with lecture-based learning (Sim + LBL).</p> Methods <p>This prospective cohort study included two consecutive undergraduate cohorts from Xiangya School of Medicine, Central South University. The 2018 cohort (<i>n</i> = 164) received traditional LBL, while the 2019 cohort (<i>n</i> = 182) was taught using Sim + LBL in the dermatology and venereology course. Both groups completed the same sexually transmitted diseases (STDs) module. Academic performance was assessed through a post-class test and an anonymous questionnaire. An eXtreme Gradient Boosting (XGBoost) model with SHapley Additive exPlanations (SHAP) was used to identify factors associated with test performance, and mediation analysis was applied to investigate the mechanisms through which the teaching model produced its effects.</p> Results <p>The Sim + LBL group achieved higher post-class scores than those in the LBL group (median difference = + 6.0, 95% CI [4.0–8.0], <i>δ</i> = − 0.39, medium effect). The XGBoost–SHAP model showed good predictive performance on the test set (<i>R</i>² = 0.82) and indicated that knowledge mastery and learning interest were the strongest contributors to academic outcomes. Mediation analysis revealed significant indirect effects of the teaching modality via knowledge mastery (Q6: <i>β</i> = 6.48, 95% CI 5.18–7.94, <i>P</i> &lt; 0.001) and learning interest (Q7: <i>β</i> = 3.16, 95% CI 2.01–4.28,<i>P</i> &lt; 0.001).</p> Conclusion <p>The Sim + LBL model was associated with better academic performance and greater student engagement by fostering cognitive and motivational gains.</p>