Detection and localization of visual anomalies are essential tasks in many safety-critical real-world scenarios. Our approach involves a vision transformer network with an association divergence, combining elements of patch embedding and a reconstruction-based method. We face challenges in capturing rich associations between abnormal patches and the entire image, as anomalies primarily focus on the adjacent patches. However, the nontrivial associations between normal patches and the entire image can be easily expressed by the self-attention weights of each patch. We use a minimax strategy to amplify this association divergence, inherently distinguishing between normal and abnormal patches and retaining the location details of the divided patches, which is subsequently handled by a Gaussian Mixture Density Network (GMDN) to pinpoint the regions of anomalies. Comprehensive experiments performed on the industrial inspection dataset show that our proposed method significantly improves visual anomaly detection and localization performance compared to competing methods.