In communities where navigating community healthcare and social services, access to clear and relevant information is more than a convenience, it is a necessity. Motivated by this need, our study explores how intelligent retrieval systems can be designed to better serve users seeking local care resources. We began with a rich directory of 1,553 community resource providers throughout Mississippi, aiming to understand how people might naturally ask for help and what ideal responses would look like. To simulate realistic interactions, we developed ten different query-reference pairs that reflect common but critical health-related concerns, ranging from mental health access to elder care services. We then implemented two information retrieval approaches: one based on Retrieval-Augmented Generation (RAG) and another using a non-RAG baseline. Using BERTScore to evaluate the semantic similarity of generated responses to our curated references, we found the RAG system to be significantly more effective, achieving an F1 score of 0.8723 compared to 0.8221 from the non-RAG model, a relative improvement of 6.1%. These results demonstrate the value of augmenting language models with curated knowledge, especially when precision and relevance matter most. Ultimately, this work highlights how thoughtful system design rooted in real-world needs can bridge the gap between resource and referral landscapes and the people who rely on them.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Comparative Analysis of RAG and Non-RAG Models to Improve Access to Service Provider Information for Older Adults in Mississippi

  • Saviz Saei,
  • Sujan Anreddy

摘要

In communities where navigating community healthcare and social services, access to clear and relevant information is more than a convenience, it is a necessity. Motivated by this need, our study explores how intelligent retrieval systems can be designed to better serve users seeking local care resources. We began with a rich directory of 1,553 community resource providers throughout Mississippi, aiming to understand how people might naturally ask for help and what ideal responses would look like. To simulate realistic interactions, we developed ten different query-reference pairs that reflect common but critical health-related concerns, ranging from mental health access to elder care services. We then implemented two information retrieval approaches: one based on Retrieval-Augmented Generation (RAG) and another using a non-RAG baseline. Using BERTScore to evaluate the semantic similarity of generated responses to our curated references, we found the RAG system to be significantly more effective, achieving an F1 score of 0.8723 compared to 0.8221 from the non-RAG model, a relative improvement of 6.1%. These results demonstrate the value of augmenting language models with curated knowledge, especially when precision and relevance matter most. Ultimately, this work highlights how thoughtful system design rooted in real-world needs can bridge the gap between resource and referral landscapes and the people who rely on them.