Scholarlink

As artificial intelligence tools are increasingly adopted to validate community service provider directories, it is critical to assess whether large language models (LLMs) can reliably verify structured data in these systems. This study evaluates five LLMs, LLaMA 3.3 70B Versatile, LLaMA 3.1 8B Instant, LLaMA 3 70B 8192, LLaMA 3 8B 8192, and Gemma2 9B IT, using community service provider data from Mississippi across three evaluation conditions: clean records (base-line), systematically corrupted entries, and records with missing fields. Model responses were categorized as “Verified,” “Not Verified,” or “Needs Checking” to assess each model’s ability to confirm correct data, reject erroneous records, and handle uncertainty, respectively. Among the models tested, LLaMA 3.3 70B Versatile demonstrated the most robust overall performance, achieving high verification accuracy on clean data (96%) and the strongest error detection capabilities by rejecting 47% of corrupted entries. In contrast, LLaMA 3 8B 8192 incorrectly verified 79% of corrupted records, indicating unsafe over-permissiveness and weak anomaly detection. These results underscore that high verification accuracy alone is insufficient; effective referral system design must prioritize models that exhibit strong error detection capabilities and appropriately defer uncertain cases to human oversight.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Beyond Accuracy: Evaluating LLMs for Validating Community Service Provider Directory

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Saviz Saei,
  • Sadhan Ghimire,
  • Sujan Anreddy

摘要

As artificial intelligence tools are increasingly adopted to validate community service provider directories, it is critical to assess whether large language models (LLMs) can reliably verify structured data in these systems. This study evaluates five LLMs, LLaMA 3.3 70B Versatile, LLaMA 3.1 8B Instant, LLaMA 3 70B 8192, LLaMA 3 8B 8192, and Gemma2 9B IT, using community service provider data from Mississippi across three evaluation conditions: clean records (base-line), systematically corrupted entries, and records with missing fields. Model responses were categorized as “Verified,” “Not Verified,” or “Needs Checking” to assess each model’s ability to confirm correct data, reject erroneous records, and handle uncertainty, respectively. Among the models tested, LLaMA 3.3 70B Versatile demonstrated the most robust overall performance, achieving high verification accuracy on clean data (96%) and the strongest error detection capabilities by rejecting 47% of corrupted entries. In contrast, LLaMA 3 8B 8192 incorrectly verified 79% of corrupted records, indicating unsafe over-permissiveness and weak anomaly detection. These results underscore that high verification accuracy alone is insufficient; effective referral system design must prioritize models that exhibit strong error detection capabilities and appropriately defer uncertain cases to human oversight.