Ranking to Learn: Human Experts, Search Engines, or LLMs for Learning Guidance
摘要
Search engines and LLMs are increasingly being used in learning contexts to find and access learning resources. While conventional ranking mechanisms in general-purpose search engines are based on topical relevance, in learning contexts, pedagogical suitability plays a crucial role in addressing learners’ information needs, i.e. how well a resource supports a learner in expanding their knowledge within a specific context. This paper conducts an empirical study, investigating how search engine rankings and LLM rankings compare to those of human experts and learners to determine which ranking approach best supports learning. Using statistical methods, we analyze agreement across rankings collected from seven experts, 60 learners, and five LLMs over four topics. Results show that LLM rankings align more closely with expert judgments than with search engines or learners. Both experts and LLMs exhibit moderate internal agreement but differ notably from search engine rankings, indicating that conventional search engines are not optimized for pedagogical effectiveness.