Compared to Unmanned Aerial Vehicles (UAVs), the Aerial Manipulator System (AMS) supports active interaction with the environment, thus having broader potential for applications. However, the motion planning of the contact-based interaction is a challenge due to the unstructured environment and the internal coupling of the AMS. The human-in-the-loop teleoperation system can enhance the intelligence of the AMS and simplify its planning layer. Current applications often separately teleoperate the UAV and manipulator, leading to a heavy cognitive burden on operators. This work establishes an all-in-one AMS teleoperation framework based on virtual reality (VR) technology. In this scheme, a digital twin model developed by Unity provides an immersive scenario in the Head Mounted Display (HMD). Additionally, with the intuitive operation methods provided by VR controllers and trackers, even one untrained operator using only one arm can guide the AMS to complete the aerial task. A experiment validated the robust collaborative capability of the proposed teleoperation system.