In this paper, we introduce a novel multimodal vision-based rehabilitation system aimed at addressing the challenges inherent in assessing hand and finger movements during physical rehabilitation. Traditionally, such assessments require individuals to be physically present in hospitals, posing inconvenience and time constraints, especially for athletes and patients recovering from hand and finger injuries. Our proposed system focuses on two specific exercises, the Claw Stretch and Thumb Stretch, known for their benefits in rehabilitation. Leveraging a hand pose estimation model with image processing techniques, our vision-based system tracks and analyzes hand and finger movements, extracting key features such as finger positions, landmark keypoints, joint angles, and hand gestures. Furthermore, we incorporate facial emotion recognition to gauge the difficulty level of exercises during rehabilitation. The primary aim is to simplify the assessment process, enabling remote monitoring by physiotherapists and granting patients the flexibility to perform exercises at home, thus eliminating the need for physical presence in traditional therapy sessions. Through testing with a sample of 20 individuals, including both males and females, we demonstrate that our approach yields assessments comparable to traditional methods, indicating its effectiveness in enhancing rehabilitation. Additionally, our results provide insights that can help therapists and patients design faster recovery measures. Users appreciate the flexibility of performing exercises in the comfort of their homes while still receiving accurate evaluations, suggesting the potential of our vision-based approach to improve accessibility and effectiveness in rehabilitation programs.