<p>While the current preference optimization methods for aligning large language models have demonstrated promising performance, extensive reliance on large amounts of manually annotated preference data presents significant barriers for broader application. The acquisition of annotated data is costly and inherently subjective, bringing challenges to the model’s fairness, robustness and trustworthiness. We propose a Self-Iterative Preference Alignment (SIPA) method, integrating both On-policy and Off-policy optimization strategies to reduce this reliance. We investigate the crucial role of self-optimization in model alignment by conducting experiments on a biased dataset and the evaluations are designed using the LLM-as-Judge framework. The experimental results indicate that our method improves the preference alignment performance of the baseline model effectively. Specifically, it achieves better performance with only ten percent human-annotated preference data, demonstrating its strong potential for application in low-resource scenarios. The data distribution shifts during iterative training are analyzed by visualizations, highlighting SIPA’s effectiveness in enhancing model alignment with human preferences.</p>