The rationality of financial behavior refers to the normative aspect of personal financial decision-making, which plays a crucial role in college students’ financial well-being. Mathematical literacy, as a key influencing factor, requires further exploration regarding its relationship with the rationality of financial behaviors and the underlying mechanisms. The aim of the present study is to explore how advanced mathematics courses impact the rationality of financial behavior among college students, with a focus on the mediating role of stability personality. A questionnaire survey was conducted among 6,065 college students in Sichuan Province, and statistical analysis was performed using the PROCESS tool. The results show that advanced mathematics courses significantly enhance the rationality of financial behavior. Furthermore, personality stability, including need for cognition, self-efficacy, and delayed gratification, partially mediates this effect. The findings emphasize the importance of mathematical literacy in improving the rationality of financial behavior and provide practical recommendations for universities to strengthen students’ financial literacy through targeted educational strategies.