Natural convection finned heat sinks are widely utilized in the nuclear field. Enhancing the convective heat transfer capability of finned heat sinks is a crucial measure to ensure reactor safety and improve efficiency. To enhance the heat dissipation capability of the heat sink, this study utilized a natural convection substitution model and the optimality criterion method for topology optimization design on traditional fin heat sink. Additionally, fractal biomimicry principles were employed to analyze the shape of the optimized structure. This study conducted topology optimization on the heat sink structure with different volume constraints. The heat transfer performance of the optimized structures under different volume constraints was analyzed through numerical simulations. The heat sink structure after topology optimization presents a leaf vein pattern, with the fins extending from the heat source gradually becoming thinner and displaying fractal features at the end of the fins. Additionally, small fins grow on the surface of the fins to further enhance convective heat transfer. Numerical simulation analysis shows that compared to traditional heat sink, the leaf vein-like heat sink is more conducive to heat conduction and convective heat transfer. Finally, the box-counting method was used to further analyze the fractal characteristics of the optimized fins, verifying their similarity to plant leaf veins, and thus demonstrating that the topology optimization structure approaches an optimal solution found in nature.