Automated radiology report generation (RRG) of routine 2D and 3D radiographs, such as X-ray and computed tomography (CT), has great potential in reducing the workload, variations, and errors of report writing and facilitating patient care. Despite significant advancements in linguistic quality, existing methods may generate reports with hallucinated type I and II (false positive and false negative) errors, which limit clinical efficiency. To mitigate the hallucinations, we propose RRG-DPO, an innovative direct preference optimization procedure with a new loss term, both tailored for effective alignment with the preference for clinically accurate RRG. RRG-DPO retrieves a set of highly relevant reports closest to the preferred response (i.e., the ground truth (GT) report) in a biomedical CLIP embedding space, and selects the one with the most significant abnormality conflicts with the GT as the dispreferred response. Besides being clinically relevant and abnormally aware, this preference data curation process is cost-effective and scalable compared to using large language models for response sampling or evaluation. In addition, we note that except for the abnormality-conflicting sentences, other sentences of the dispreferred report can legibly describe the radiograph of the preferred in a clinically equivalent manner, despite variations in expression. Thus, RRG-DPO creates a sub-preferred report from the dispreferred by deleting the abnormality-conflicting sentences, and promotes its likelihood with a new loss term. RRG-DPO is evaluated on both 2D X-ray and 3D CT data to align a wide range of RRG models. Experiments show that it boosts the clinical efficiency of all assessed models in six metrics: precision, recall, F1 score, RadGraph, RadCliQ, and RaTEScore, effectively reducing hallucinations. Further ablation studies show that our method outperforms DPO and DPOP, and its components are helpful.