Recent performance improvement of the Chinese Spelling Check (CSC) has shown the effectiveness of introducing a non-parametric memory component to retrieve similar examples. Nevertheless, most existing approaches only use this memory during testing, which hinders the memory retriever from adapting to the specific CSC model and leads to sub-optimal error correction. Furthermore, these approaches suffer from inefficient inference since they perform memory retrieval on every position throughout an entire sentence. This paper proposes a framework called Retrieve-or-Copy (RoC) to enhance the standard CSC model with an error detector, a learnable retriever, and a character-level copy mechanism. Specifically, in RoC, the error detector identifies incorrect characters before retrieval, enabling the retriever to focus solely on errors. By jointly optimizing the retriever and the CSC model with spelling error correction as the ultimate training objective, the retriever is more likely to find suitable examples for the CSC model to achieve better performance. Additionally, the copy mechanism allows the CSC model to access the correct character in the input to avoid over-correction. Experimental results demonstrate that RoC outperforms methods that only use non-parametric memory during testing and achieves comparable or even better performance than state-of-the-art CSC models on SIGHAN datasets. Our code is publicly available at https://github.com/dingdingtom/csc_roc .