Despite the great performance of recent LLMs across various tasks, their tendency to hallucinate poses significant risks for real-world applications. Prior studies have attempted to mitigate hallucinations through external knowledge supplementation, while with substantial computational expenses and dependence on high-quality knowledge sources. In this study, we introduce a novel decoding strategy, POSITIVE (decoding with part-of-speech (POS)-adapTIVE crItics), in order to enhance the factuality of LLM. Concretely, we assume that tokens with different lexical functions exhibit distinct relationships with hallucinations. After investigating the relation between different POS words and hallucinations on each layer in Transformers, layer-wise critics were developed to detect potential hallucinations for different POS words. These critics dynamically monitor and intervene during the decoding phase, guiding the model to generate more factual responses. Experimental results indicate that the proposed POSITIVE significantly improves factuality without resorting to external resources or models. This research contributes a novel perspective to the exploration of controlled generation mechanisms grounded in syntactic or semantic attributes.