In the context of intelligent technologies empowering learning environments, cultivating students’ collaborative problem-solving (CPS) skills through effective teamwork to address complex tasks has become an integral component of the 21st-century core competencies framework. However, the diagnosis of CPS skills, as the primary step in personalized development, faces numerous challenges. Traditional “human-to-human” assessment methods lack sample independence, while “human-to-computer” assessment methods, although overcoming this limitation, suffer from issues of unrealistic scenario construction. To address this challenge, this study focuses on the assessment of CPS skills in a human-to-computer collaborative setting, leveraging the superior performance of generative AI in text understanding and reasoning. Based on a multi-agent technology framework and using the PISA 2015 assessment approach as a foundation, this study constructs an agent collaboration environment powered by generative AI technology to simulate highly realistic human-to-human collaborative scenarios. This method aims to accurately evaluate students’ actual levels of collaborative problem-solving skills. From the perspectives of the construction and implementation of multi-agent collaborative scenarios and the measurement of CPS skills, this study provides new ideas and implementation pathways for educational assessment empowered by generative AI.