Visual information is widely used in peg-in-hole assembly due to the features that can quickly reduce the uncertainty of the position and the lack of contact with the target hole surface. This paper proposes a peg-in-hole assembly method based on hybrid visual information. Firstly, we acquire 3D point cloud data, and based on the RANSAC algorithm, we obtain the direction vectors of the holes, to calculate the required orientation of the end-effector. Then we propose a two-stage target detection network that can directly extract the position of the target hole, and to alleviate the large amount of time required to collect and label the real images, we train on the synthetic dataset and part of the real dataset obtained from the constructed virtual assembly scene. The method was trained and tested on the constructed simulation platform, proving that the assembly strategy can be effectively applied to robot peg-in-hole assembly.