Autonomous exploration is an important method for obtaining the environment map. In this paper, we propose an autonomous unmanned aerial vehicles (UAVs) exploration method in unknown environments using octomap. Firstly, a frontier detection algorithm based on octomap is proposed to increase detection efficiency. Secondly, we propose a hybrid frontier clustering method combining DBSCAN and k-means for viewpoint generation. Thirdly, an asymmetric travelling salesman problem is formulated for exploration path planning. Finally, we evaluate our method in two types of scenes in the simulation environment. The simulation results show that our method can achieve autonomous exploration of 1000 m3 space within 412 s and build a fine map efficiently.