Scholarlink

The rapid adoption of Large Language Models (LLMs) is transforming diverse sectors, including education, by enabling personalized learning experiences. Artificial Intelligence (AI) agents support adaptive instruction, instant feedback, automated assessment, and content generation, enhancing access to individualized resources. However, current systems often lack accuracy and offer limited Human–Agent Interaction (HAI), a key factor in effective student engagement. To address this, we propose a visual interaction mechanism that augments chatbot communication with image-based responses. We extend an LLM-powered tutor agent by incorporating a Large Action Model (LAM), enabling the agent to execute image-based functions alongside text generation. Building on a LangGraph-orchestrated prototype, we developed two LAM-integrated tools: 1) an Image Retrieval Tool that accesses a vector database of textbook illustrations, and 2) an Image Cropping Tool that dynamically extracts regions from textbook pages. These were implemented using OpenAI’s Function Calling API and evaluated with the GPT-4o mini model. In controlled trials on single textbook pages, the Image Retrieval Tool consistently identifies relevant illustrations, while the Cropping Tool frequently returns visually plausible but pedagogically irrelevant regions. This reveals a precision–flexibility trade-off: curated retrieval ensures accuracy, whereas dynamic cropping offers broader applicability at the expense of reduced relevance. We also identify challenges in synchronizing tool calls with textual explanations and scaling manual dataset preparation. Our results suggest that LAM-enhanced function-calling can significantly improve HAI in educational contexts, highlighting directions for automating content curation and enhancing interaction consistency.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Augmenting an LLM-Based Tutor Agent with a Large Action Model for Multimodal Interaction

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Piotr Kluczyński,
  • Arthur Picard,
  • Yazan Mualla,
  • Hedi Tebourbi,
  • Sana Nouzri,
  • Abdeljalil Abbas-Turki,
  • Franck Gechter

摘要

The rapid adoption of Large Language Models (LLMs) is transforming diverse sectors, including education, by enabling personalized learning experiences. Artificial Intelligence (AI) agents support adaptive instruction, instant feedback, automated assessment, and content generation, enhancing access to individualized resources. However, current systems often lack accuracy and offer limited Human–Agent Interaction (HAI), a key factor in effective student engagement. To address this, we propose a visual interaction mechanism that augments chatbot communication with image-based responses. We extend an LLM-powered tutor agent by incorporating a Large Action Model (LAM), enabling the agent to execute image-based functions alongside text generation. Building on a LangGraph-orchestrated prototype, we developed two LAM-integrated tools: 1) an Image Retrieval Tool that accesses a vector database of textbook illustrations, and 2) an Image Cropping Tool that dynamically extracts regions from textbook pages. These were implemented using OpenAI’s Function Calling API and evaluated with the GPT-4o mini model. In controlled trials on single textbook pages, the Image Retrieval Tool consistently identifies relevant illustrations, while the Cropping Tool frequently returns visually plausible but pedagogically irrelevant regions. This reveals a precision–flexibility trade-off: curated retrieval ensures accuracy, whereas dynamic cropping offers broader applicability at the expense of reduced relevance. We also identify challenges in synchronizing tool calls with textual explanations and scaling manual dataset preparation. Our results suggest that LAM-enhanced function-calling can significantly improve HAI in educational contexts, highlighting directions for automating content curation and enhancing interaction consistency.