The existing multi-task dense prediction methods based on Mixture of Experts (MoE) have achieved impressive performance. However, these methods often suffer from limited diversity and specialization among experts. In this paper, we propose a novel group sparse regularization approach for MoE-based multi-task dense prediction, termed Mixture of Group Experts (MoGE). MoGE indirectly regularizes experts by imposing structure constraints on the inputs to the top-k routing, thereby preserving the original MoE architecture. We further introduce intra-task and inter-task regularization terms to encourage similar inputs and related tasks to activate similar sets of experts, thereby significantly enhancing expert diversity and specialization. Extensive experiments on the PASCAL-Context and NYUD-v2 benchmarks demonstrate that our MoGE achieves state-of-the-art performance with negligible additional memory and computational overhead. The source code is available at:  https://github.com/wangyuankl123/MoGE-for-MTDP .