Hallucination is a persistent challenge in large-scale language models, manifesting at multiple stages and leading to outputs that stray from reality and produce some content that does not conform to common sense. We introduce a novel approach to alleviate hallucination by contrasting the probability of intermediate layer with the last layer to obtains the next-token distribution during inference. Then, introduce in-layer stability factor to tackle the issue of token probability fluctuation across transformer layers. Our approach effectively addresses the issue of inconsistent output distributions from lower decoder layers in extensive models, evidenced by impressive results on benchmarks such as GSM8k, StrategyQA, and Wiki Factor. These outcomes highlight the significant potential of our method in reducing hallucination in large language models.