This paper presents the design and optimization of a compact quasi-optical (QO) mode converter for a high-performance gyrotron operating at 105 GHz in the \(TE_{17,6}\) mode. The converter integrates a dimpled-wall launcher with a novel dual-direction perturbation technique alternating positive and negative deviations based on coupled mode theory. This approach reduces the launcher length to 85 mm, with a cut length of 20 mm (23.53% of the total length), while maintaining high mode conversion efficiency. MATLAB-based parametric analysis was used to optimize the launcher’s field distribution, and FEKO simulations validated its radiation performance. The mirror system comprising quasi-elliptical, elliptical, and parabolic mirrors ensures precise phase correction and beam shaping, contributing to high mode purity and compactness. Simulations demonstrate outstanding performance, achieving 99.4% scalar and 98.6% vector Gaussian mode content at the output. This design offers a compact and efficient solution for next-generation millimeter-wave applications.