Recent advances in Large Language Models (LLMs) have shown impressive capabilities in programming tasks, prompting our interest in applying LLMs in Data-Flow Analysis (DFA). DFA is essential for compiler optimization and cybersecurity. Given the inherently graph-based nature of the traditional DFA algorithm and LLMs’ emerging graph reasoning abilities, this paper investigates their effectiveness in DFA tasks from a graph reasoning perspective. However, our evaluations report that direct prompting, providing only a task description and code sample, often leads to poor results, as LLMs struggle with stepwise reasoning just like the traditional DFA algorithm To address this, we propose involving structured algorithmic explanations and formalized reasoning examples in prompts, showing that well-crafted prompts significantly improve performance. We also identify result collapse in LLM’s reasoning process, where early-stage errors propagate and distort final outputs. To mitigate result collapse, we propose error-case-augmented prompting, which explicitly incorporates common mistakes and corrections into the prompt. Compared to baseline strategies such as iterative prompting, our experiments demonstrate that error-case augmentation is crucial for enabling LLMs to perform accurate DFA reasoning. This study provides new insights into leveraging LLMs for DFA reasoning and highlights the importance of tailored prompting techniques to enhance their reliability in program analysis.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Prompting Large Language Models for Data-Flow Analysis: Addressing Common Errors for Better Reasoning

  • Qingchen Yu,
  • Jianshan Zhang,
  • Chunming Wu

摘要

Recent advances in Large Language Models (LLMs) have shown impressive capabilities in programming tasks, prompting our interest in applying LLMs in Data-Flow Analysis (DFA). DFA is essential for compiler optimization and cybersecurity. Given the inherently graph-based nature of the traditional DFA algorithm and LLMs’ emerging graph reasoning abilities, this paper investigates their effectiveness in DFA tasks from a graph reasoning perspective. However, our evaluations report that direct prompting, providing only a task description and code sample, often leads to poor results, as LLMs struggle with stepwise reasoning just like the traditional DFA algorithm To address this, we propose involving structured algorithmic explanations and formalized reasoning examples in prompts, showing that well-crafted prompts significantly improve performance. We also identify result collapse in LLM’s reasoning process, where early-stage errors propagate and distort final outputs. To mitigate result collapse, we propose error-case-augmented prompting, which explicitly incorporates common mistakes and corrections into the prompt. Compared to baseline strategies such as iterative prompting, our experiments demonstrate that error-case augmentation is crucial for enabling LLMs to perform accurate DFA reasoning. This study provides new insights into leveraging LLMs for DFA reasoning and highlights the importance of tailored prompting techniques to enhance their reliability in program analysis.