Information extraction (IE) is a pivotal task in natural language processing (NLP), aiming to extract structured information from unstructured text. However, extracting domain-specific information that focuses on a particular subject area is challenging since many existing solutions lack adaptability to new or evolving domains, compromising models’ efficacy and scalability. Recent achievements in pre-trained language models (LM), particularly large language models (LLM), have shown strong capabilities in context understanding, making them a promising solution for IE tasks based on a generative paradigm. To explore the performance of LM-based IE approaches, we used four different pre-trained language models, from traditional BERT model to open sourced LLMs, to extract domain-specific information based on a Chinese military news dataset. We compared the precision, recall, and f1-score of different approaches under different situations, such as few-shots, prompt tuning, and LoRA-based fine-tuning. Empirical results demonstrate that context learning could, to some extent, improve the performance of LLM-based IE solutions. Nonetheless, challenges such as limited adaptability, the misalignment of structured outputs, and constraints imposed by limited token sizes continue to impact the performance of LM-based IE solutions, especially for relation extraction (RE) tasks.