Modern deep learning models require large amounts of labeled data and high computational power, especially for tasks like object detection that involve complex scenes and many object classes. These challenges become even more difficult in multi-label settings, where the model needs to detect multiple overlapping objects from different categories. Previous research has explored using graph structures to represent relationships between classes, and Mixture of Experts (MoE) to divide work across specialized sub-models. In this paper, we combine both ideas by introducing a Graph-based Mixture of Experts (GMoE) architecture for multi-class object detection. Our method groups related object classes into smaller subsets and assigns each to a dedicated expert model. A graph built using class co-occurrence data helps decide which experts should be used for each image, reducing unnecessary computation. We benchmark our architecture on a custom version of the COCO 2017 dataset, utilising RetinaNet with a ResNet-50 FPN backbone. The architecture displays a strong performance on large and complex scenes, achieving a mean Average Precision (mAP) of 0.7431 on the “Outdoor” class, while smaller objects like “Kitchen” items remain challenging at 0.1484 mAP. Although focused on object detection, this architecture can be applied to other tasks where speed, accuracy, and modular design are important.