The research of depth estimation in panoramic images can improve the accuracy of depth estimation in real world. However, according to the survey, most of the current work is based on ordinary planar images, and only a few studies are carried out based on panoramic images. Due to the lack of panoramic image depth data set containing both indoor and outdoor scenes, the depth estimation of these panoramic images is almost based on indoor panoramic images, and the existing methods can not solve the unified estimation of panoramic image depth of indoor and outdoor scenes. In addition, the problems of complex feature extraction, image distortion and wide field of view multi-scale information extraction have not been effectively solved. Therefore, PanoDth, a panoramic image depth dataset containing both indoor and outdoor, is constructed in this paper to provide basic conditions for the research of indoor and outdoor depth unified estimation network model. We propose the PanoDthNet, which features two innovative designs, the panoramic feature aggregation(PFA) module and the panoramic multi-scale attention (PMSA) module. PFA is used to improve the feature extraction ability of panoramic images and to deal with image distortion. PMSA can effectively acquire wide field of view and multi-scale information. A large number of experiments show that PanoDthNet performs better than the existing methods on the PanoDth dataset. The relative accuracy measures ( \(\delta <\) 1.25) reached 89.52%.