In recent years, Vision-Language Models (VLMs) have demonstrated remarkable capabilities in scene understanding, common sense reasoning, and decision-making in the field of autonomous driving, bringing new opportunities for the development of autonomous driving technology. However, existing methods typically integrate VLMs directly into the decision loop, resulting in high inference latency that severely affects the system’s real-time response and driving safety. To address this issue, we propose a semi-decoupled system architecture that decouples VLMs from the real-time control loop, allowing them to function as asynchronous mid-term planners. Furthermore, to ensure efficient and stable collaboration between the high-latency asynchronous VLM decisions and the high-frequency underlying vehicle control, we design a three-layer planning system consisting of macroscopic global routes, VLM mid-term target points, and microscopic PID controllers. Additionally, we introduce a memory module to enhance the decision robustness in rare scenarios. Experimental results in the CARLA simulator demonstrate that the system reduces control layer latency by at least 25% (to 42 ms from the baseline of 56 ms), while maintaining high decision accuracy and cross-scenario generalization ability across various driving tasks.