One of the most common tasks in histopathology is the visual comparison of the images of successive multiply stained tissue sections. Automatic image registration is crucial to perform this analysis. Although the tissue sections in general undergo non-rigid deformations, the initial linear image alignment impacts the overall registration drastically. However, most of the recent works do not study the linear transformation compensation separately and focus on the non-linear part. In this work, we propose a novel unsupervised feature matching approach for affine registration of histological images. We perform the evaluation on the Automatic Non-rigid Histological Image Registration (ANHIR) dataset and show the supremacy of our method over the existing affine registration approaches in therms of accuracy and robustness. The code is available at https://github.com/VladPyatov/UnFeMa .