Fiber-optic transmission systems (FOTS) play a crucial role in establishing broadband communication channels across telecommunications networks, offering high data rates and reliability. In modern transport communication infrastructures, Optical Transport Hierarchy (OTH) transmission systems are the most widely adopted due to their efficiency in handling large volumes of data traffic. This paper introduces a novel approach to determining the optimal repeater span length in FOTS that utilize quadrature amplitude modulation (QAM), a technique known for improving spectral efficiency. To achieve this, an advanced simulation model is developed within the MatLab environment, designed to comprehensively analyze the performance of OTH systems with various QAM-M modulation levels. The study explores how adjusting signal power at the transmitter output can enhance transmission efficiency and extend the repeater span length. Results demonstrate that for an OTU3 optical channel, the maximum achievable repeater span length of 1500 km is obtained using QAM-4 modulation. This finding emphasizes the potential for substantial improvements in long-haul communication by optimizing modulation techniques and transmitter power levels, making FOTS even more viable for extensive broadband network deployments.