Aligning LLMs with human preferences is widely recognized as a foundational factor in their recent advancements. This alignment process often necessitates substantial quantities of annotated data, the collection of which is both time-intensive and laborious when relying solely on human annotators. To mitigate this, previous work has introduced the concept of LLM-as-judge. However, this approach has limitations, including the considerable financial costs, privacy leakage, challenges in reproducibility caused by frequent model updates, and potential biases inherent in proprietary systems. An alternative line of research has explored fine-tuning open-source models using feedback data generated by proprietary models like GPT-4 to improve alignment capabilities. While these methods have demonstrated effectiveness, they typically depend on large-scale synthetic training datasets. To address these challenges, we propose Probe Guided Decoding (PGD), a novel approach that enhances alignment capabilities while requiring only a minimal number of training samples and simple linear classifiers. PGD consistently surpasses the LLM-as-judge paradigm across diverse settings and significantly mitigates potential biases on two publicly available datasets with different LLMs. Notably, on the JudgeBench benchmark, PGD achieved a remarkable 11.8% improvement with as few as 10 training samples, compared to LLM-as-judge.