Reference-based super-resolution (RefSR) utilizes an external high-resolution reference (Ref) image to transfer detailed textures to a low-resolution (LR) image, resulting in improved performance over single-image super-resolution (SISR) methods. The main challenge in RefSR is to find correspondences between the LR and Ref images, and accurately convey the rich texture information of the Ref image. However, this becomes difficult when the similarity between the LR and Ref images is low or there is ambiguity in the matching stage. To address these challenges, we propose a novel cross-and self-feature transformer (CSFT) which integrates not only the rich visual features of the Ref image, but also the internal information within the input LR image. In addition, we introduce a high-frequency feature alignment (HFFA) module to robustly fuse the features of the LR and Ref images even in areas where alignment is ambiguous. Based on the proposed CSFT and HFFA modules, we define a new RefSR pipeline, referred to as CSSR, where each module is structured with multi-scales. The CSSR can fully utilize textural information in both Ref and LR images and achieve outstanding performance, even when feature matching between Ref and LR images is challenging. Various experiments have been conducted to verify the effectiveness of CSSR, both quantitatively and qualitatively. The source codes is available at https://github.com/SeonggwanKo/CSSR .