LLMs for Question-Answer and Synthetic Data Generation and Evaluation
摘要
Question-answer generation (QAG) focuses on creating answers to a set of questions based on a pre-specified context. Some applications for QAG are included in the field of information retrieval and data augmentation for QA models. However, one of the most popular uses of automatic answer retrieval presently is notably in the field of education. This study focuses on harnessing advanced artificial intelligence (AI), in particular, large language models (LLMs) for creating a Question-Answering system by generating answers for a set of predefined questions in healthcare. This sub-system will feed into and be integrated into a larger framework for a project with a goal of building a bilingual (English and Spanish) intelligent tutoring system (ITS) geared towards educating survivorship skills to low-literacy Hispanic breast cancer survivors. Several Hugging Face’s Models and evaluation metrics were used for generating and evaluating answers based on previously-generated health-related questions. Synthetic data generation and Code-switching functionality was also incorporated to determine how well the models perform when accounting for both English and Spanish. GPT-generated English answers achieved higher accuracy than Spanish answers (96.38% vs 86.11%) when manually evaluated. The manual evaluation was based on grammar, correctness, and meaningfulness based on the health information transcriptions and questions. Automated answer evaluation for code-switched English and Spanish scores differed by 0.71% between human and AI automated evaluation. Within the different aspects of experimentation for this study, the usefulness and applicability of generative AI for virtual tutoring purposes was continually proven, and provides ground for further discussion of how this constantly-emerging technology could be used to benefit diverse learning populations.