MoE-Reasoner: Towards a Balanced Language Model
摘要
Large language models (LLMs) have shown impressive capabilities across various domains, particularly in mathematical reasoning and code generation. Advances in deep learning and the scale of pretraining have enabled LLMs to tackle complex tasks that require logical deduction, multi-step calculations, and program synthesis. However, models fine-tuned for enhanced reasoning often incur significant computational costs and may suffer from reduced performance on tasks outside their primary domain. This work addresses these challenges by proposing a mixture-of-experts (MoE) architecture with a reasoner model, that dynamically routes queries to specialized experts, thereby achieving a balance between high task performance and computational efficiency. The approach leverages a reasoning-focused expert based on the deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B model, alongside a code expert trained with LoRa adapters. Experimental results demonstrate that this MoE-reasoner achieves strong, balanced outcomes on both mathematical and coding benchmarks, outperforming models specialized for either domain alone. The proposed design represents a promising step toward the development of adaptable, multitask-capable LLMs suitable for real-world applications where both versatility and efficiency are paramount.