The 6-DoF grasp detection typically relies on the point cloud, which provide the 3D representation of the environment; or the RGB images, which offer global color information. To utilize both spatial and color information, existing methods apply RGB-D images to generate grasping; however, these methods fail to exploit the valuable 3D representation, compromising the grasp quality with sub-optimal solutions. This paper proposes a novel grasp detection method named the Fusion GraspNet, which effectively utilizes the fused features from point cloud and an RGB image in the 3D space to improve grasp quality. To achieve this, a Layer-Level Spatial-Aware Fusion U-Net framework is constructed that (i) extracts the futures of point could and RGB image, and (ii) aligns and fuses the point cloud and RGB features in layer level, effectively preserving the 3D spatial representation while capturing the global 2D color information. In addition, a point sampling and view selection mechanism is developed that further utilizes the fused features to refine the grasp label distribution. Experiments based on the GraspNet-1Billion benchmark show that the proposed method outperforms the state-of-the-art by over 7 AP (Average Performance), with real-world experiments further validating its effectiveness.