Emotional decline is a gradual and often overlooked process among older adults, with serious consequences for mental health and quality of life. Existing emotion detection methods typically rely on physiological sensors or isolated speech recordings. However, these approaches lack temporal continuity and are unsuitable for long-term monitoring. To address these limitations, we propose CEDME (Continuous Emotion Decline Monitoring for the Elderly), a multimodal and temporally structured framework that continuously monitors emotional well-being from naturalistic daily speech. CEDME captures fine-grained vocal and lexical signals through gated fusion, and models emotional trajectories over time via hierarchical temporal inference across both intra-day and cross-day scales. To support this effort, we construct a Chinese elderly speech dataset (CESD) comprising 14 consecutive days of conversational recordings from older adults in China. Extensive experiments on long-range emotional monitoring show that CEDME consistently outperforms strong baselines, confirming its effectiveness as a scalable, non-intrusive, and clinically interpretable solution for real-world monitoring of emotional decline in the elderly.