Reliable and convenient simulation platforms are essentially important for advancing robotic technologies quickly, especially for drones which are with high security risks. Aimed to provide a high-efficiency public simulation platform for various aerial manipulation tests, this paper presents a modular aerial manipulation system (AMS) based on a simulation platform named XTDrone. The robot for aerial manipulation consists of a quadrotor platform and a five-degree-of-freedom robotic arm with a gripper jaw in the end. A segmented control strategy is used to separately control the Unmanned Aerial Vehicle (UAV) and the robotic arm. Additionally, the system architecture as well as its core configuration is described in details in this paper. A visual servo algorithm is employed to enhance object recognition capabilities. Finally, to validate the effectiveness and feasibility of the proposed system, experiments on specific aerial manipulation tasks are conducted to simulate and operate the aerial manipulation robot under visual servo control.