In the monarchy of human–machine interaction or human–computer interaction, the Kinect emerges as a usual interaction device operating within the paradigm of regular consumer interface. With its low-cost accessibility and SDK solutions, Kinect has addressed numerous challenges. However, persistent issues include sensitivity to light variations in the tracker due to a color-based point search model. The current system design grapples with non-homogeneous backgrounds, challenges distinguishing cloth color from skin color, inadequate segmentation, static gestures, limited depth information, and the constraint of single-user recognition. In response, this research delves into developing a novel algorithm for multiple skeleton recognition, specifically focusing on the segmented portions of the human body. Further, it can achieve more accuracy and precision during communication between humans and computers.