Tribal and agro-based rural livelihoods are seriously threatened by the regular natural disasters caused by climate change. Risk perceptions are crucial for planning and catastrophe avoidance. Addressing and preventing catastrophes and maladaptation are hampered by a lack of analytical evidence on how vulnerable people perceive climate hazards. This study uses a mixed methodology to evaluate how families perceive climate change and analyze how it affects their livelihoods in Lahaul and Spiti, Himachal Pradesh, India, which faces significant climate-related difficulties. To analyze the susceptibility of livelihood due to climate change, the Intergovernmental Panel on Climate Change (IPCC) framework for the Livelihood Vulnerability Index (LVI) is used. The data was collected from both primary and secondary sources. For the three contributing factors of LVI for Lahaul (10 panchayats), Spiti (7 panchayats), and Udaipur (7 panchayats) blocks, eight major components were selected, of which there are 46 distinct sub-indicators at the Panchayat level, the LVI was computed, in with the village level data aggregated. The highly exposed block is Spiti, followed by Lahaul, and the lowest is Udaipur. Within the panchayat, the most exposed in Spiti are Khurik, Kaza, and Kibber; in Lahaul, Jobrang, Darcha, and Yurnath are highly exposed; and in Udaipur the highly exposed panchayats are Udaipur, Tindi, and Nalda. The studied households observed declining water sources, occasional droughts, livestock losses, and unpredictable weather conditions. The research area’s livelihood vulnerability was found to be correlated with climatic variability, natural disasters, health, food, and social factors.