The computation of outage probability (OP) and ergodic rate of the non-orthogonal multiple access (NOMA) systems over the generalized multipath fading channel has been extensively studied in the literature. In this paper, it is shown that the conventional signal-to-noise ratio (SNR)-based approach overestimates the OP of the cooperative NOMA users for the modulation order \(M>4\) of unequal energy signal constellation. This result is due to the invalid assumption that the OP is independent of the transmitted symbols and the power allocation scheme does not depend on the modulation order of the near user. To address this issue, a constellation-based approach is proposed for the OP computation, and the condition for the power allocation is obtained from the NOMA signal constellation using M-ary square quadrature amplitude modulation (QAM). Through analytical and simulation results, the impact of inaccuracy in the OP computation of the SNR-based approach is investigated on the performance of the successive interference cancellation receiver at the near user and the coverage probability at the relay node. Furthermore, an upper bound for the transmission rate of the NOMA users is derived based on the power allocation constraint obtained from the NOMA signal constellation.
