In this chapter, the convergence control for achieving the stability of first order reliability method (FORM)-based steepest formulation is provided by improved HL-RF iterative formulation. Three conditions as (i) merit condition, (ii) angle condition and (iii) sufficient descent condition are presented for controlling instability of the FORM-based improved HL-RF and relaxed FORM approach. These conditions are applied for a suitable framework for monitoring the stability of FORM-based steepest descent method which are dynamically applied for adjusting step size. The step size using an accelerated inexact inner loop is computed for achieving the stable results of iterative FORM formulated by steepest descent methods.