Despite the great progress of anomaly detection technology, the mainstream anomaly detection (AD) methods still face the challenge of accurate detection of semantic anomalies. In this work, we propose a novel AD framework, to mitigate this problem. We introduce two specially-designed modules: Input-Reference Alignment (I-RA) and Adaptive Multi-scale Ensembled Scoring (A-MES). In I-RA, one ORB (Oriented Fast and Rotated Brief)-based spatial alignment block is introduced to constrain the patch matching from feature-only to a consistent measure in both feature and location, which can make the position-related semantical anomaly, such as wrong-printed letters in garment printings, be detected more easily. In order to against large variance of anomaly scale, A-MES is also developed to generate patches of different scale and multi-scale-fused abnormal scores, so that the detect performance for both the small-scale defects and semantic anomalies can be increased further. On the widely used MVTec AD dataset and our specially constructed Garment Printing Defects (GPD) dataset, our method achieves performance comparable to or even better than SOTA, especially in semantic anomalies and small-scale anomaly detection.