Portfolio management refers to the strategic process of selecting and allocating financial assets to optimize returns while effectively managing risks. It encounters significant challenges in adapting to financial markets, managing nonlinear financial risks, and achieving sample efficiency in dynamic environments, which can result in poor robustness and decreased long-term performance. To address these challenges, this paper presents Adaptive Portfolio Management Policy Optimization (APMPO), a novel framework that, to the best of our knowledge, is the first to integrate neural stochastic differential equations within a Transformer architecture enhanced by meta-learning-inspired fast adaptation. APMPO introduces a multi-objective optimization scheme incorporating volatility-penalized advantage calculations and employs adversarial training combined with optimized experience replay for enhanced robustness. Comprehensive experiments conducted on five years of Dow Jones Industrial Average (DJIA) constituent stock data demonstrate that APMPO consistently outperforms existing methods, achieving at least 8.97 \(\%\) higher absolute cumulative returns and 24.9 \(\%\) higher Sharpe ratios than its competitors. These results suggest that APMPO provides a reliable and innovative solution to modern portfolio management challenges.