Due to the distortion of projection generated during the production of \(360^{\circ }\) video, most quality assessment algorithms used for 2D video have the problem of performance degradation. In this paper, we propose a full-reference \(360^{\circ }\) video quality assessment method, utilizing saliency to guide viewport extraction to eliminate the projection distortion. To be more specific, we first predict the visual saliency of each frame with a \(360^{\circ }\) saliency prediction network and then select the viewport that optimally represents the video frame through the optimal viewport positioning module (OVPM). Furthermore, we propose the attention-based three-dimensional convolutional neural network (3D CNN) quality assessment network to evaluate the video quality, in which 3D CNN convolution and attention modules can better capture the quality degradation of distorted viewports. Experimental results show that our method achieves superior performance in \(360^{\circ }\) video quality assessment tasks.