The ultimate bearing capacity ( \({Q}_{u}\) ) of rock-socketed piles is determined by the combined end-bearing resistance at side resistance and the pile tip along the pile shaft. This research innovatively addresses the complex challenge of assessing pile tip-bearing capacity in rock formations, a task that traditionally relies on intricate numerical models often inaccessible for routine engineering. By establishing an Adaptive Neuro-Fuzzy Inference System (ANFIS) model and hybrid models incorporating the Flying Fox Optimizer (FFO) and Coot optimization algorithm (COA), this study offers a novel approach to accurately calculating the \({Q}_{u}\) of rock-socketed piles, making the process more efficient and practical for engineers. According to similar research, a number of factors were chosen as inputs to combine the impacts of soil and rock layers, including Uniaxial compressive strength of pile (UCS), total length ( \(p\) ) to diameter (D), the length of the soil layer ( \({L}_{s}\) ) to socket length ( \({L}_{r}\) ), Standard penetration test (N_SPT), and reference depth in rock layers ( \({H}_{r}\) ). Selected performance metrics, such as RAE, RMSE, MAE, R2, and PI, were employed for identifying the suitable models and evaluate the results after these systems were developed and modeled. The analysis revealed that the hybrid ANCA (ANFIS + COA) and ANFF (ANFIS + FFO) models significantly outperform the conventional ANFIS model. The ANCA model’s performance is especially noteworthy; it showed a 1.8% gain in the coefficient of determination (R2) and an astounding 43.6% reduction in the root mean square error (RMSE) when in comparison with the ANFIS model. Accurate \({Q}_{u}\) estimates optimize foundation design, reducing material wastage and construction costs while mitigating risks associated with foundation failure. This precision also aids in meeting regulatory standards and minimizing environmental impact by avoiding excessive excavation.