Reinforcement learning (RL) and imitation learning (IL) are two prominent frameworks for training control policies in robotic systems. While RL enables autonomous policy acquisition through environmental interaction, it often suffers from low sample efficiency and requires carefully crafted reward functions. On the other hand, IL leverages expert demonstrations but typically assumes near-optimality, which is not always guaranteed in practice. Recently, learning from human preferences has emerged as a complementary approach that enables agents to improve policies based on relative assessments of behavioral quality, rather than absolute rewards or optimal samples. However, this approach alone is insufficient when both compared demonstrations are poor. In this study, we propose a unified framework that integrates RL, IL, and preference-based learning. Our method first initializes a policy and reward function via generative imitation learning using suboptimal demonstrations. The learned policy is then used to generate new trajectories, which are scored using the learned reward. A human annotator provides preferences over these trajectories, and when the human feedback contradicts the prior score, the policy is refined via preference learning. Discrepant samples are also recycled into the imitation loop. We validate our approach on two real-robot bimanual manipulation tasks and show that it achieves superior data efficiency compared to baseline methods.