<p>In recent years, researchers have focused on improving the efficiency of geothermal heat exchangers. This is crucial in advancing renewable energy and reducing greenhouse gas emissions. Research gaps in the design of helical geothermal heat exchangers include the determination of a suitable helix geometry and using hybrid nanofluids. Therefore, it is necessary to perform numerical simulations in this field to find the thermal performance of the above-mentioned modifications. This study presents a new geometry of a conical helical heat exchanger with half the length of the tubes in similar works that gives the same outlet temperatures in the range of ± 0.02%. To do so, because the soil temperature increases with the depth of the soil, the pitch of the coil in the lower parts of the heat exchanger is reduced. Also, the number of revs of the heat exchanger was reduced and the length of the heat changer tube was reduced by 48%. For the simulation of heat transfer and fluid flow in the pipe, a 3D model simulation was performed in a steady state and laminar flow inside the heat exchanger using CFD. Finally, different volume fractions of Al₂O₃, SiO₂, and hybrid nanoparticles 50% Al₂O₃ and 50% SiO2 were added to the optimized geometry to enhance the heat transfer rate. The results showed that in laminar flow with constant inlet velocity, by adding nanoparticles to water with a volume fraction of 3%, the convection heat transfer coefficient increased by 8.5%, 5%, and 1.6% for Al₂O₃, hybrid Al₂O₃ + SiO₂, and SiO₂ nanoparticles, respectively. Also, for the cases with constant Reynolds number, the convection heat transfer coefficient increased by 8.2%, 5.8%, and 3.5% for Al₂O₃, hybrid Al₂O₃ + SiO₂, and SiO₂ nanoparticles, respectively.</p>