<p>The Heat Recovery Steam Generators (HRSGs) are designed to recapture heat from gas turbine exhaust to generate electric power by a steam turbine. Finned tubes are critical heat transfer components of HRSG, whose design and optimization play an essential role in realizing effective energy utilization in power plants. In this study, an optimization method with multi-layered neural network (MNN) and tree-structured parzen estimator (TPE) is proposed for the finned tube heat exchanger with curved serration. This method has high fitting accuracy and global optimization efficiency, and is suitable for complex heat transfer design and optimization problems caused by novel irregular finned tubes. The developed thermal-fluid model is validated with existing experimental data, and a satisfactory agreement is found in terms of Nusselt number and Fanning friction factor. It is shown that increasing the curved serration angle is beneficial to destroy the thermal boundary layer and enhance turbulent kinetic energy. When the Reynolds number is between 5000 and 25 000, the heat transfer factor of finned tubes with a curved serration angle of 10° is 20% higher than that of flat serrated finned tubes on average. The optimized geometric parameters are obtained from the optimization approach, and the optimal solution has achieved excellent results in comprehensive performance. Compared with the baseline design, the optimized results show a 9% higher heat transfer factor, which is better than those based on commonly used optimization methods.</p>