Autonomous decision making across tasks without dedicated tuning is a notable cognitive ability of large language model (LLM) agents. However, it is also widely observed that LLM agents may experience serious hallucinations in long-term decision tasks which require actions over a long horizon. To generate decision actions, most approaches leverage complete decision history as context or summarize the long decision history into shorter context. Nonetheless, this approach results in highly context-sensitive decisions inevitably. In this paper, we first transform the decision history into visual diagrams and propose a novel dual stream long-term decision framework for LLM agents. Specifically, we adopt three visual diagrams (flowchart, mindmap, and sequence diagram) to represent three key ingredients (task plan, exploration, and operation) of decision making, respectively. Then, we devise the visual reflection mechanism that utilizes the text failure process to update the visual flowcharts and mindmaps, forming visual memory which are directly used in the subsequent iteration to help LLM agents get rid of hallucinations. Our method harnesses the excellent text and visual reasoning ability of LLMs and offers LLM agents a complementary visual perspective to enhance the traditional text-based decision making process. Extensive experiments conducted in two interactive environments have shown that our method helps LLM agents overcome hallucinations and improves performance on both tasks.