This study investigates the potential of integrating passive intelligent omni surface (IOS) technology to enhance wireless communication within a hybrid non-orthogonal multiple access (NOMA) system. Through a comprehensive analysis, three user pairing schemes - near-near, far-far (NN-FF) pairing, odd-even (O-E) pairing, and near-far (N-F) pairing - are meticulously investigated. Analytical expressions for outage probability are derived for each of the three pairing schemes. The proposed IOS-assisted hybrid NOMA with N-F pairing demonstrates a significant signal-to-noise ratio (SNR) improvement, achieving gains of \(\sim\) 48 dB to \(\sim\) 52 dB compared to reconfigurable intelligent surface (RIS)-assisted hybrid NOMA and conventional hybrid NOMA systems. The N-F pairing strategy excels with the IOS-assisted hybrid NOMA, delivering superior performance over other pairing strategies. However, it results in a complex system, whereas NN-FF pairing exhibits a low-complexity system. Furthermore, the power allocation factors are optimized to maximize the sum capacity across all users. This work also explores the impact of imperfect successive interference cancellation (iSIC) on system performance. The proposed IOS-assisted hybrid NOMA system with N-F pairing demonstrates significant improvements in sum capacity. The N-F pairing achieves a minimum of \(\sim\) 4% increase in sum capacity over other pairing strategies, even with a 1% SIC error rate. The proposed schemes with optimal power fractions improve sum capacity (bps/Hz) from \(\sim\) 8.64 to \(\sim\) 10.46 compared to conventional systems. The proposed system demonstrates significant improvements in sum capacity and outage probability performance, providing essential insights for the advancement of next-generation networks.
