Optimizing the performance of heat exchangers (HEs) can help supervise a significant amount of energy in a different of industrial applications. The many advantages of FTHEs, such as a higher area of heat transfer, have led to their rise in popularity over conventional HEs in the last few decades. The geometry of the tubes has a significant influence on the performance and drop in pressure in these equipment. Several investigations have been conducted to improve the thermal efficiency of FTHEs with hexagonal, elliptical, and circular tubes by numerically and experimentally. To enhance an FTHE’s performance, four different octagonal tubes with various side ratios are used in the current study. At varying Reynolds numbers, the capacity of heat transfer and fluid flow properties of the FTHE are investigated numerically with respect to varied octagonal side ratios. The results are showed that the octagonal tube HE with a side ratio of 0.5 and a plain-fin design performed the best. In comparison with HEs with circular or hexagonal tubes, FTHEs have a lower pressure drop, which gives them an advantage even if their side ratio is 0.5 and they have the lower values for the Colburn factor and heat transfer.