Evaluating chatbots in the LLM era: a hybrid survey and qualitative synthesis of evaluation frameworks
摘要
Large language models have rapidly transformed conversational AI, creating a pressing need for evaluation methods that go beyond traditional text similarity or translation-based metrics. Modern chatbots demand assessment frameworks capable of measuring complex dimensions such as multi-turn coherence, reasoning quality, factual accuracy, alignment with safety constraints, and robustness under adversarial conditions. This survey provides a comprehensive synthesis of current evaluation methodologies, spanning rule-based scoring, automated metrics, learned evaluators, LLM-as-a-judge systems, consensus-based multi-model evaluation, safety and adversarial stress-testing, and human-in-the-loop protocols. We propose an eight-part taxonomy to categorize the evolving landscape of evaluation techniques, analyze their strengths and limitations, and highlight emerging reliability trends. In addition, we compile and characterize major datasets, benchmarks, and open-source tools used in contemporary chatbot evaluation. Persistent challenges—including bias, hallucination detection, long-context handling, and scalable human evaluation—are identified as critical research gaps. To address these limitations, we outline a conceptual unified evaluation pipeline that integrates multiple assessment paradigms. This work aims to serve as a foundational reference for future research toward more reliable, transparent, and trustworthy evaluation of LLM-driven conversational agents.