Scholarlink

<p>As developers increasingly rely on screenshots to communicate programming issues on platforms like Stack Overflow, there is a growing need to understand how generative AI models can support this shift from text-based to visual problem descriptions. This study investigates two key capabilities of large language models (LLMs) in this context. First, we assess whether LLMs can infer the intent and content of a programming query directly from a screenshot, without relying on textual input. Second, we evaluate their ability to detect the presence of relevant text within these images and to extract that content accurately. We evaluated three state-of-the-art LLMs—LLAMA, GEMINI, and GPT-4o—using prompt-based techniques such as in-context and few-shot learning, ap-plied to a manually curated dataset of 223 Stack Overflow posts with screenshots. For question inference, GPT-4o achieved the strongest results, with over 60% similarity to the original question for 51.75% of images. Developer-rated relevance reached 0.69, and embedding-based similarity peaked at 0.59, particularly for screenshots with focused, unambiguous content like code or error messages. For relevance detection and text extraction, we evaluated model performance with and with-out OCR assistance. GEMINI achieved the highest F1 score (0.91) and recall (0.98), performing consistently across both conditions, indicating robust visual understanding. GPT-4o benefited sig-nificantly from OCR input, reaching BLEU 0.46 and ROUGE-1 0.68, while LLAMA lagged behind due to formatting inconsistencies and lower extraction quality. Our findings highlight both the potential and current limitations of LLMs in supporting image-driven software support. They also point to concrete next steps, including targeted fine-tuning for visual programming tasks and improved prompt strategies for mixed-content inputs.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Do multimodal LLMs understand programming screenshots? Inferring questions and extracting relevant content

  • 期刊论文

  • 发表日期：

    2026年3月9日

查看全文
  • Faiz Ahmed,
  • Xuchen Tan,
  • Folajinmi Adewole,
  • Suprakash Datta,
  • Maleknaz Nayebi

摘要

As developers increasingly rely on screenshots to communicate programming issues on platforms like Stack Overflow, there is a growing need to understand how generative AI models can support this shift from text-based to visual problem descriptions. This study investigates two key capabilities of large language models (LLMs) in this context. First, we assess whether LLMs can infer the intent and content of a programming query directly from a screenshot, without relying on textual input. Second, we evaluate their ability to detect the presence of relevant text within these images and to extract that content accurately. We evaluated three state-of-the-art LLMs—LLAMA, GEMINI, and GPT-4o—using prompt-based techniques such as in-context and few-shot learning, ap-plied to a manually curated dataset of 223 Stack Overflow posts with screenshots. For question inference, GPT-4o achieved the strongest results, with over 60% similarity to the original question for 51.75% of images. Developer-rated relevance reached 0.69, and embedding-based similarity peaked at 0.59, particularly for screenshots with focused, unambiguous content like code or error messages. For relevance detection and text extraction, we evaluated model performance with and with-out OCR assistance. GEMINI achieved the highest F1 score (0.91) and recall (0.98), performing consistently across both conditions, indicating robust visual understanding. GPT-4o benefited sig-nificantly from OCR input, reaching BLEU 0.46 and ROUGE-1 0.68, while LLAMA lagged behind due to formatting inconsistencies and lower extraction quality. Our findings highlight both the potential and current limitations of LLMs in supporting image-driven software support. They also point to concrete next steps, including targeted fine-tuning for visual programming tasks and improved prompt strategies for mixed-content inputs.