Multimodal Large Language Models (MLLMs) have achieved significant progress recently on a variety of tasks. By combining pre-trained visual encoders with large language models, MLLMs can perform multimodal tasks such as describing images, visual question answering based on visual inputs and language instructions. Despite their success, MLLMs still suffer from the issue of hallucination, where the output generated by the model appears coherent and reasonable but is in fact inconsistent with the visual content. In this work, we treat the problem of hallucination reduction as a human preference alignment issue. Specifically, instead of using any human or expert models for data annotation, we construct preference data pairs by using the outputs of model with fewer hallucinations as preferred responses, and the outputs of models subjected to image distortions as non-preferred responses. The models are then trained by direct preference optimization to learn the subtle differences between the responses with varying degrees of hallucinations, and thus prefer non-hallucinatory responses. This is an automated approach that has a very low costs while being easily scalable. Experimental results show that our method is effective in mitigating hallucinations.