Large-Scale Language Models (LLMs) fundamentally excel at next token prediction. In contrast, Named Entity Recognition (NER) is a sequence labeling task, where models must assign entity-type labels to each token. This fundamental difference presents challenges for LLMs in NER tasks. Recent research has shown that LLMs with over 100 billion parameters have made progress in NER through in-context learning, designed to bridge this gap. However, this method has not fundamentally addressed the negative effects of the paradigm difference. More importantly, in-context learning relies on LLMs with parameters exceeding 100 billion, requiring substantial computing resources and indicating that its strategy is not optimal. In our study, we propose CoT-NER to align the paradigms of LLMs and NER. We introduce chain of thought (CoT) into LLMs, which transform NER into a less-step reasoning task and bridges the gap between traditional sequence labeling tasks and next token prediction. In addition, in order to further enhance the reasoning ability of LLMs, we propose a retrieval strategy based on entity-distribution, which provides positive demonstration when we combine CoT with in-context learning. And we propose semi-automatic method for data balancing. By fine-tuning LLMs with 8B-14B parameters efficiently on CoT, we enhance inference ability while reducing computational resources. We conducted experiments on widely used NER datasets. The results indicate that CoT-NER achieves better performance compared to current mainstream LLMs. Importantly, our method demonstrates superior performance in limited-resource scenarios with extremely scarce training data, showing CoT-NER’s potentiality for real-world NER applicotations with limited labeled samples.