Large language models (LLMs) have shown impressive capabilities across various domains, particularly in mathematical reasoning and code generation. Advances in deep learning and the scale of pretraining have enabled LLMs to tackle complex tasks that require logical deduction, multi-step calculations, and program synthesis. However, models fine-tuned for enhanced reasoning often incur significant computational costs and may suffer from reduced performance on tasks outside their primary domain. This work addresses these challenges by proposing a mixture-of-experts (MoE) architecture with a reasoner model, that dynamically routes queries to specialized experts, thereby achieving a balance between high task performance and computational efficiency. The approach leverages a reasoning-focused expert based on the deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B model, alongside a code expert trained with LoRa adapters. Experimental results demonstrate that this MoE-reasoner achieves strong, balanced outcomes on both mathematical and coding benchmarks, outperforming models specialized for either domain alone. The proposed design represents a promising step toward the development of adaptable, multitask-capable LLMs suitable for real-world applications where both versatility and efficiency are paramount.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MoE-Reasoner: Towards a Balanced Language Model

  • Alan-Barsag V. Gazzaev,
  • Valentin A. Malykh

摘要

Large language models (LLMs) have shown impressive capabilities across various domains, particularly in mathematical reasoning and code generation. Advances in deep learning and the scale of pretraining have enabled LLMs to tackle complex tasks that require logical deduction, multi-step calculations, and program synthesis. However, models fine-tuned for enhanced reasoning often incur significant computational costs and may suffer from reduced performance on tasks outside their primary domain. This work addresses these challenges by proposing a mixture-of-experts (MoE) architecture with a reasoner model, that dynamically routes queries to specialized experts, thereby achieving a balance between high task performance and computational efficiency. The approach leverages a reasoning-focused expert based on the deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B model, alongside a code expert trained with LoRa adapters. Experimental results demonstrate that this MoE-reasoner achieves strong, balanced outcomes on both mathematical and coding benchmarks, outperforming models specialized for either domain alone. The proposed design represents a promising step toward the development of adaptable, multitask-capable LLMs suitable for real-world applications where both versatility and efficiency are paramount.