Despite excelling at common-sense reasoning, large language models (LLMs) still lag behind humans in multi-turn reasoning within interactive environments. This gap arises from four key limitations: unreliable reasoning, hallucinations, memory decay, and the opacity of closed-source models. To address these challenges, we propose the Aware-of-subPlans via Supervised Fine-Tuning (AoPSFT) optimization paradigm, which features three key innovations: (1) a transparent, reproducible, and safer learning paradigm based on the open-source LLaMA model; (2) a structured task decomposition mechanism that harnesses the inherent planning capabilities of LLMs to break tasks into subgoals; and (3) enhanced step reasoning through explicit subgoal awareness during interactions, enabling dynamic goal adjustment via real-time feedback. This framework ensures that the agent explicitly tracks subgoal progress at each interaction step, mitigating reasoning opacity while improving interpretability. Empirical evaluations demonstrate that AoPSFT achieves an average score of 78.68 on unseen ScienceWorld tasks with an 8B-parameter model, surpassing previous competitive open-source algorithms by 18.2%. In addition, our work provides a reproducible pathway for developing efficient and interpretable interactive agents. Code and dataset are available at: https://github.com/Sunrepe/AoPSFT