The supercritical CO2 Brayton cycle (SCBC) boasts advantages in efficiency and size, presenting broad prospects for its application in nuclear power ships. In contrast to typical optimization scenarios, the system design of shipboard SCBC generally has access to system-level parameter constraints, but is devoid of state constraints for individual equipment points. Due to these intractable prior constraints, this paper presents a data-driven-assisted method under uncertain boundary conditions for the system optimization. First, a thermodynamic model for a typical regenerative SCBC is established, including turbomachinery, heat exchangers, and a nuclear heat source. For the system design under uncertain boundaries, a data-driven-assisted optimization method that integrates genetic algorithm (GA) optimization with Q-learning for boundary exploration is developed. Within this framework, the boundaries of the parameters to be optimized are adaptively adjusted based on the optimization results of GA. According to the dynamic boundaries, the states of the working fluid in various parts of SCBC can achieve the maximum possible overall system matching. Meanwhile, two populations are mixed within GA, facilitating information interactivity between the original boundaries and the updated boundaries. In addition, to mitigate computational costs, the offline model of SCBC is constructed using the back propagation neural network. The optimization results demonstrate that even when the initial boundaries do not encompass the global optimum, adaptive boundaries can effectively search for the optimal parameter range dynamically. The cycle efficiency and power density of the designed SCBC system can be approximately 50% and 800 kW·m−3 simultaneously, achieving global optimization under the adaptive boundaries.