ProKG-Dial: A Progressive LLM-Driven Approach to Building Knowledge-Intensive Multi-turn Dialogue Datasets with Domain Knowledge Graphs
摘要
Current large language models (LLMs) excel at general NLP tasks but often lack domain-specific precision in professional settings. Building a high-quality, domain-specific multi-turn dialogue dataset is essential for developing specialized conversational systems. However, existing methods—such as manual annotation, simulated human-LLM interactions, and role-based LLM dialogues—are resource-intensive or suffer from limitations in dialogue quality and domain coverage. To address these challenges, we introduce ProKG-Dial, a progressive framework for constructing knowledge-intensive multi-turn dialogue datasets using domain-specific knowledge graphs (KGs). ProKG-Dial leverages the structured nature of KGs to encode complex domain knowledge and relationships, providing a solid foundation for generating meaningful and coherent dialogues. Specifically, ProKG-Dial begins by applying community detection to partition the KG into semantically cohesive subgraphs. For each subgraph, the framework incrementally generates a series of questions and answers centered around a target entity, ensuring relevance and coverage. A rigorous filtering step is employed to maintain high dialogue quality. We validate ProKG-Dial on a medical knowledge graph by evaluating the generated dialogues in terms of diversity, semantic coherence, and entity coverage. Furthermore, we fine-tune a base LLM on the resulting dataset and benchmark it against several baselines. Both automatic metrics and human evaluations demonstrate that ProKG-Dial substantially improves dialogue quality and domain-specific performance, highlighting its effectiveness and practical utility.