Postoperative upper limb dysfunction is prevalent among breast cancer survivors, yet their adherence to at-home rehabilitation exercises is low amidst limited nursing resources. The hardware overhead of commonly adopted VR-based mHealth solutions further hinders their widespread clinical application. Therefore, we developed Breast-Rehab, a novel, low-cost mHealth system to provide patients with out-of-hospital upper limb rehabilitation management. Breast-Rehab integrates a bespoke human action recognition algorithm with a retrieval-augmented generation (RAG) framework. By fusing visual and 3D skeletal data, our model accurately segments exercise videos recorded in uncontrolled home environments, outperforming standard models. These segmented clips, combined with a domain-specific knowledge base, guide a multi-modal large language model to generate clinically relevant assessment reports. This approach significantly reduces computational overhead and mitigates model hallucinations. We implemented the system as a WeChat Mini Program and a nurse-facing dashboard. A preliminary clinical study validated the system’s feasibility and user acceptance, with patients achieving an average exercise frequency of 0.59 sessions/day over a two-week period. This work thus presents a complete, validated pipeline for AI-driven, at-home rehabilitation monitoring.