<p>Object goal navigation, which involves autonomous navigation towards an object goal in an unseen environment, is a fundamental task for embodied AI agents. Recent works tackle this task mainly through solely reinforcement learning (RL) based methods or large language model (LLM) based methods. However, RL-based methods rely solely on historical trial-and-error experience, suffering from poor generalization to new environments. And LLM-based methods fall short in terms of reliable decision-making and cost-efficiency. To address these challenges, we propose RLLMNav, a method that integrates historical experience of RL with commonsense reasoning of LLM through a confidence-gated routing mechanism. Rather than replacing either paradigm, RLLMNav coordinates them so that each compensates for the other’s failure modes. Specifically, we utilize commonsense knowledge extracted from an LLM to suggest frontiers. We also use this form of knowledge in a dual-module strategy to select frontiers from suggestions as long-term goals to explore, where an LLM-based policy for accessing LLM commonsense knowledge improves generalization, and an RL-based policy for learning historical experience improves reliable decision-making and cost-efficiency. Extensive experiments on Gibson and Habitat-Matterport 3D (HM3D) demonstrate that RLLMNav achieves state-of-the-art results, validating the potential of combining RL and LLMs for ObjectNav. Code is available at <a href="https://github.com/ckx666/RLLMNav">https://github.com/ckx666/RLLMNav</a>.</p>