Compared with the standalone operation, unmanned aerial vehicle (UAV) swarm systems bring the high efficiency to the task execution in a cooperative manner, and they has gained the widespread application in disaster relief, power inspection and other fields. To support the fulfillment of cooperative task, the cooperative positioning capability plays the key role in global positioning system (GPS) denied areas. However, there are still difficulties in initialization, high communication demand, and inevitable drift for swarm cooperative positioning. In order to solve the problems above, a distributed collaborative simultaneous localization and mapping (SLAM) system is developed based on ranging measurement from ultra-wideband (UWB), which specifically accelerates the initialization by introducing prior information, and optimizes the communication framework between the UAVs so as to run in real time. Besides, the ranging is introduced as the restraints on the relative drift between the nodes. The evaluation are performed on datasets recorded in a lab environment. Experiments results show that our method surpasses 15% over the mainstream visual-inertial odometry in positioning accuracy. Furthermore, our method effectively reduces the drift of the visual-inertial odometry in challenging scenarios on visual.