This paper introduces a real-time Human–Robot Interaction (HRI) system that establishes a gesture-based interface between a patient and a healthcare robot. The system detects patient motion through image processing, extracting geometric features from the patient’s hand movements. The system also incorporates real-time environmental monitoring using an Android camera, facilitated by an attendant in situations where a patient’s condition is too severe for the self-controlling of the robot. In scenarios necessitating social distancing, such as the COVID-19 pandemic, the proposed system holds the potential to control a healthcare robot. This system, incorporating computer vision technology and an Android camera, empowers the robot to securely grasp objects like medication, hand sanitizer, monitoring equipment, or food through a gripper control mechanism. The gesture recognition system accurately identifies right-hand gestures 93.75% of the time and left-hand gestures 91.25% of the time.