The assembly of small to medium-sized complex products in fields such as aerospace, automotive, and industrial robotics directly impacts product performance and quality. Although human-robot collaborative assembly provides an effective solution for the assembly of such products, uncertainties in the actual process consistently pose challenges to the system’s stability. To address the challenge of capturing subtle actions that result in operator uncertainty, an end-to-end skeleton-RGB video feature integrated model focusing on operator-part interaction is developed for highly similar human action online prediction. Secondly, to address the issue of disorganized placement and occlusions in assembly parts that lead to uncertainty in the collaborative robot’s performance, an intent-guided target part perspective focusing and grasping pose estimation approach is proposed. Thirdly, to address diverse uncertainties caused by human actions and robotic grasping, a dual-agent adaptive control approach based on human and grasping environment perceptions is proposed. Finally, taking a reducer assembly as an example, the proposed approaches are validated on a hardware platform. It is expected to reduce the cycle time of collaborative tasks from the closed-loop perspective of perception, prediction, decision, and control.