Mobile edge computing (MEC) enhances computation efficiency and reduces latency by offloading tasks from mobile devices (MDs) to nearby edge servers (ESs). To optimize the offloading process and manage resource allocation effectively, deep reinforcement learning (DRL) has been widely adopted as a promising solution. However, as the number of MDs increases, the rapidly expanding state-action space poses significant challenges to the ability of DRL to learn effective policies, often resulting in suboptimal decision-making. Large language model (LLM) with strong reasoning capabilities and extensive prior knowledge offers a potential solution by enabling more efficient exploration and guiding the DRL toward better policies. Therefore, we propose an LLM-guided soft actor-critic (LLM-guided SAC) algorithm, which integrates LLM-generated policy priors with a probabilistic mixed strategy to facilitate learning in high-dimensional decision spaces. By refining the state-action representation, the proposed algorithm enhances policy learning efficiency, while LLM-guided priors enable informed exploration and improve early-stage decision quality. Moreover, the mixed strategy balances exploration and exploitation, contributing to stable and effective learning. Experimental results demonstrate that LLM-guided SAC consistently outperforms baseline methods, particularly in large and complex decision spaces, highlighting its strong potential for resource optimization in MEC networks.