<p>A new technique, steel pile-large geotextile cofferdam, is employed for the cofferdam on soft clay as a cost-effective and structurally stable solution for construction projects. This study delves into the failure mechanisms and stability quantification of steel pile-large geotextile cofferdam constructed on typical soft-over-stiff soil profiles through comprehensive numerical simulations. The simulations have been rigorously validated against existing data, demonstrating a high level of accuracy. A detailed parametric analysis was carried out to investigate the key factors affecting the cofferdam’s failure mechanism, including soil properties, steel pile dimensions, and geotextile bag dimensions, further quantifying the critical fill height and overall stability during both construction and operational phases under seepage load. The results indicate a significant positive influence of steel piles on cofferdam stability. A novel formula with a high degree of accuracy (<i>R</i>² = 0.85) is developed to predict the stability of cofferdam under seepage conditions, offering valuable insights for the design and construction of this innovative cofferdam technology.</p>