RGB-Thermal (RGB-T) semantic segmentation presents a promising approach for achieving robust environmental perception under challenging lighting and adverse weather conditions. However, this task remains challenging due to insufficient generalization capability and limited adaptability to dynamic environmental conditions. While Vision Foundation Models (VFMs) offer powerful feature representations, their direct application to RGB-T segmentation is hindered by scale limitations in dense prediction tasks and inflexible fusion strategies. Therefore, we propose MFESeg, a novel framework that bridges these gaps through two key innovations: 1) A Multi-Scale Adaptation (MSA) module that injects hierarchical visual priors into frozen VFMs via parameter-efficient low-rank decomposition and dilated convolutions, enhancing segmentation capability while minimizing trainable parameters; 2) A Mixture of Fusion Experts (MoFE) module that dynamically activates optimal fusion pathways through multi-modal routing, enabling context-aware integration of complementary RGB-T features. Extensive experiments on MF and PST900 datasets demonstrate state-of-the-art performance, with MFESeg achieving 61.9% mIoU on MF dataset and 89.9% mIoU on PST900 dataset while requiring few trainable parameters. Our work establishes an effective paradigm for adapting large-scale VFMs to RGB-T semantic segmentation task.