In optimizing CO \(_2\) capture and sequestration, accurate phase behavior prediction in subsurface reservoirs is critical. This is especially relevant for geologic CO \(_2\) storage in deep saline aquifers and depleted oil/gas reservoirs, and for CO \(_2\) -EOR, where injected CO \(_2\) coexists with brine and resident hydrocarbons across wide pressure–temperature ranges (including near-critical CO \(_2\) ) and with non-negligible mutual solubilities. Our work introduces an efficient algorithm for phase stability testing, combined with multistage equilibrium multiphase negative flash calculations. We explore the stability of a three-phase system (V-L-Aq), using the criterion that the system is stable if the phase fractions of vapor, liquid, and water phases ( \(\beta _V\) , \(\beta _L\) , \(\beta _W\) ) fall between 0 and 1. The stability testing begins with a negative flash calculation using diverse initial K-values to detect multiple phases. The same criterion is applied to two-phase systems (V-L, V-Aq, L-Aq). In case of instability, the algorithm shifts to the most probable two-phase system by systematically using negative flash and tangent plane distance calculations. It determines stable phases based on tangent plane solutions when the negative flash fails to converge, demonstrating adaptability across multiphase systems. The algorithm is validated through simulations of various mixtures, showing reliable convergence near critical points, and is applied to CO \(_2\) storage and EOR scenarios to predict different CO \(_2\) trapping mechanisms in the subsurface.