This paper introduces a method for calculating sub-zero temperature airflow and heat transfer within the RELAP5 framework. Utilizing this approach, a hybrid cooling passive residual heat removal system is designed, employing sub-zero temperature air and a cooling water tank as the ultimate heat sink. Compared with the experimental results of natural cold source heat exchanger, the temperature difference of shell side outlet water is only 0.08 K. With these improvements, RELAP5 can calculate air flow and heat transfer processes at sub-zero temperatures. Transient analysis results of air–water mixed cooling passive residual heat removal system indicate that mixed cooling system can lower the average temperature of primary system to the NRC safety threshold within 36 h after station break out accident. With water cooling accounting for 31% of the total decay heat, the cooling tank's water usage is minimized, ensuring system safety and long-term core cooling stability.