<p>Physical hardening of asphalt materials subjected to prolonged low-temperature conditions results from the densified shrinkage and increased stiffness caused by molecular rearrangements. It significantly contributes to early low-temperature cracking in asphalt pavements, leading to further distress and raising maintenance expenses. Several engineering dosages (3%, 4%, and 5%) of SBS modified asphalt binders were prepared to clarify the influence of physical hardening on their low-temperature performance. Extended bending beam rheometer test, molecular dynamics simulation, and atomic force microscope were conducted to explore the rheological characteristics and hardening mechanism. Statistical methods were performed to establish the characteristic correlation between macromechanics and micromorphology, and some specific evaluation criteria were proposed. The results show that the higher modifier content indicates the more pronounced deterioration of rheological properties subjected to hardening and a higher risk of low-temperature cracking in service, similar to a longer hardening time and lower temperature. The motion of asphalt molecules exhibits an annealing acceleration state followed by a relaxation steady state during the hardening process. The higher modifier content and lower temperature lead to severe molecular aggregation, while the temperature is the main factor compared to the modifier content. The “bee” structure transits from an integral decomposition stage to a local aggregation stage where the altitude and size grow as the SBS content increases. Grade loss is recommended as the criterion for the hardening characteristics assessment of SBS modified asphalt binders. Mean area and the range between peaks and valleys (<i>R</i>_max) can depict the microstructural properties under the influence of hardening time and SBS content, respectively.</p>