The Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) task aims to match a given query sketch with the corresponding real image, which has important practical applications. Traditional methods mainly use a triplet network framework based on triplet loss and adopt CNN as the backbone network. This method relies on obtaining triplet annotations of challenging sketches and positive/negative sample images. In this paper, we introduce the BYOL framework from contrastive learning and propose a FG-SBIR method called Sketch-BYOL that does not require negative samples. To address the diversity of sketch shapes and the small scale of FG-SBIR datasets, we design a backbone network combining deformable convolutions and ViT, which surpasses traditional CNN networks in feature representation extraction capability and can be directly trained on small-scale datasets. We have experimentally demonstrated the effectiveness of the Sketch-BYOL method.