<p>Efficiently modelling nonlocal pixel dependencies is the key factor for boosting the image restoration performance. We find that most Transformer-based models mix the local and nonlocal information in self-attention (SA) layer. However, a pixel is more closely related to its neighbours than to distant pixels. Furthermore, existing research demonstrated that large self-attention weights would become larger and larger during the training process. Therefore, the local and nonlocal mixed SA tends to rely on local features rather than nonlocal information, reducing their ability to model nonlocal pixel dependencies. To address this problem, we propose a novel Transformer design direction for image restoration, which for the first time attempts to remove the local pixel dependencies in SA layer and compensate for local information in the feedforward network layer. Based on this direction, we present an efficient Transformer with only nonlocal information-modeled SA (ONLformer) for Image Restoration via several simple designs, i.e. hourglass attention (HA), nonlocal top-k (NLTK) sampling, and dual separate feedforward Network (DSFN). The proposed model achieves favorable performance in comparisons with the state-of-the-art methods and incurring low computational costs on four representative image restoration tasks, including image deraining, dehazing, deblurring and denoising.</p>