Automatic Question Answering for the Linguistic Domain – An Evaluation of LLM Knowledge Base Extension with RAG

We investigate the extent to which Retrieval Augmented Generation improves the quality of Large Language Models’ answers to technical questions in the field of linguistics—a domain known for its broad terminological inventory and theory-dependent use of technical terms. Furthermore, this application is not only about terminological information on language, but also about information on its well-formedness. We present the results of an empirical evaluation of automatically generated answers based on authentic data from a language consulting service, with special emphasis on different question types.