Recently, the widespread adoption of Large Language Model (LLM) technologies has drawn significant attention. Both enterprises and individuals increasingly leverage these Artificial Intelligence (AI) systems across various services, particularly through chatbot interfaces. This study evaluates the current landscape of publicly accessible AI chatbots by comparing their performance and ability to assess the quality of their own generated content. We examine five widely used free-access chatbots: ChatGPT-3.5, Jamba, Phind, Perplexity, and Coral, using three question-answering datasets and one text summarization dataset. These chatbots are evaluated across nine metrics, four of which are assessed manually, while the remaining five employ AI-assisted evaluation using ChatGPT-3.5, Jamba, Microsoft Copilot, and Google Gemini. Our findings position ChatGPT-3.5 as the leading general-purpose model, a result that reflects its advanced development and maturity. AI-assisted metrics demonstrate strong performance across text comprehension and knowledge-based tasks, though results in mathematical reasoning tasks were inconsistent. Human analysis indicates that AI-assisted evaluations may inflate ratings, which reveals their potential bias favoring positive sentiment in current Large Language Models. This observed bias highlights well-documented limitations in text-based AI systems, warranting further research.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Performance and Self-evaluation of Free-Access Large Language Model Chatbots: A Multi-metric Comparative Study

  • Cameron Miskell,
  • Khaled Slhoub,
  • Neji Mensi

摘要

Recently, the widespread adoption of Large Language Model (LLM) technologies has drawn significant attention. Both enterprises and individuals increasingly leverage these Artificial Intelligence (AI) systems across various services, particularly through chatbot interfaces. This study evaluates the current landscape of publicly accessible AI chatbots by comparing their performance and ability to assess the quality of their own generated content. We examine five widely used free-access chatbots: ChatGPT-3.5, Jamba, Phind, Perplexity, and Coral, using three question-answering datasets and one text summarization dataset. These chatbots are evaluated across nine metrics, four of which are assessed manually, while the remaining five employ AI-assisted evaluation using ChatGPT-3.5, Jamba, Microsoft Copilot, and Google Gemini. Our findings position ChatGPT-3.5 as the leading general-purpose model, a result that reflects its advanced development and maturity. AI-assisted metrics demonstrate strong performance across text comprehension and knowledge-based tasks, though results in mathematical reasoning tasks were inconsistent. Human analysis indicates that AI-assisted evaluations may inflate ratings, which reveals their potential bias favoring positive sentiment in current Large Language Models. This observed bias highlights well-documented limitations in text-based AI systems, warranting further research.