In response to the declining quality of user service in cell-free massive MIMO networks due to an increase in communication traffic, this paper explores the multi-domain resource management problem utilizing deep reinforcement learning. The resource management issue is formulated as a total transmission rate maximization problem, constrained by various hardware conditions while considering the distinct characteristics of access point (AP) devices and user service quality requirements. For efficient resource allocation across multiple domains, we introduce a deep reinforcement learning-based approach. Furthermore, this paper demonstrates the effective collaboration among multiple APs through a training mechanism designed to promote cooperation among the agents. Simulation results indicate that the proposed multi-domain joint resource allocation algorithm significantly improves system spectral efficiency when compared to existing resource allocation algorithms in cell-free massive MIMO system.