Sky view factor (SVF) is closely related to outdoor thermal comfort in residential areas. Existing studies typically analyze the relationship using the Universal Thermal Climate Index (UTCI) as a single indicator to evaluate outdoor thermal comfort, lacking a comprehensive consideration of sunlight and wind environment. In this paper, the wind-thermal environment of residential areas is regarded as an integrated system. Using a residential area in Xiamen as a case study, a multi-objective optimization method was applied to obtain the optimal scenarios by taking the three wind-thermal environment indicators as the joint optimization objectives. Additionally, the control strategies of SVF under the weights of different wind-thermal environment indicators were proposed. The research results show that: (1) SVF has the most significant impact on outdoor thermal comfort in summer (0.99), followed by sunlight hours in winter (0.78) and wind-thermal environment (−0.52); (2) When improving outdoor wind-thermal environments, the relationship between sunlight hours and thermal comfort should be prioritized and balanced; (3) Moderately decreasing SVF is the primary strategy to improve the overall outdoor wind-thermal environment. This study helps to reveal the reason for the complexity exhibited by SVF in evaluating outdoor comfort and to provide a trade-off for the enhancement of the outdoor wind-thermal environment in residential areas.