This paper proposes an innovative method to control the robot arm, which is based on the gesture recognition library–Mediapipe and OpenCV. The project is based on the RobotStudio platform, which simulates the development of the project. The research closely combines human–computer interaction with robot control systems. Through computer vision technology and machine learning, the system can accurately recognize human gestures and convert them into precise movements of robotic arms. This project improves the efficiency and practicality of medical treatment, industrial safety and industrial automation on the road simply and easily. The system processes gestures and generates robotic control commands via OpenCV and Mediapipe, then communicates with the robotic arm via TCP/IP and controls the robotic arm via RAPID. The system has a modular design that could be expanded to control other robotic arms and more complex gestures in the future. The project demonstrates the technology for gesture-driven robots and demonstrates the potential for future remote control, human–computer interaction, and virtual reality environments to work together.