Finding reliable correspondences from two images of the same scene is an important component in computer vision. Due to viewpoint transformations in two views and differences between the images, the initial putative matches often contain a large number of outliers, which significantly impacts the effectiveness of downstream tasks. In this paper, we propose a handcrafted outlier filtering method named Coarse-to-Fine Locally Affine Matching (CF-LAM). First, coarse filtering is designed to calculate an adaptive threshold based on the distribution characteristics within a reliable iteration, in order to remove outliers with large deviations. Then, fine filtering is performed to calculate the outlier scores of the retained correspondences, based on residuals and rankings in each iteration. Finally, the consistency of the neighborhoods is calculated to preserve the final correspondences. We have conducted experiments on viewpoint transformations datasets in different scenes, compared with state-of-the-art methods, and achieved promising performance results.