Artificial intelligence and machine learning revolutionizing the domain of fluid mechanic due to their precise modeling, optimization, and understanding the complex and nonlinearity more efficient. The author uses the AI-based Levenberg–Marquardt Scheme with a Backpropagation Neural Network (LMS-BPNN) to investigate the flow stability of MHD boundary layer flow of Casson Hybrid Nanofluid (CHNF) over a porous shrinking sheet. The partial differential equations (PDEs) that describe Casson hybrid nanofluid are transformed into a system of ordinary differential equations (ODEs) with efficient similarity variables. The initial/reference solution is generated using bvp4c function (an embedded MATLAB function designed to solve systems of ODEs) for various input parameters as demonstrated in scenarios 1–5. There are three options to divide numerical data: \(80\%\) for training, \(10\%\) for testing, and an additional \(10\%\) for validation. The LMS-BPNN is used to obtain the approximate solution for scenarios 1–5. The effectiveness and reliability of the proposed LMS-BPNN are validated through fitness curves based on correlation index ( \(R\) ), error, and regression analysis. It is noted that velocity and temperature profiles satisfy boundary conditions asymptotically for Senario1-5 with LMS-BPNN. Intelligent algorithms are used to calculate the dual solution for evaluating flow performance results. The perturbation scheme is applied to an unsteady boundary layer problem to obtain the eigenvalues problem. An unsteady solution \(f(\eta , \tau )\) converges to steady solution \({f}_{o}(\eta )\) for \(\tau \to \infty\) when \(\gamma \ge 0\) . However, an unsteady solution \(f(\eta , \tau )\) diverges to a steady solution \({f}_{o}(\eta )\) for \(\tau \to \infty\) when \(\gamma <0\) . It is found that the boundary layer thickness for the second (lower branch) solution is higher than the first (upper branch) solution. This investigation is the evidence that the first (upper branch) solution is stable and reliable. The analysis of errors demonstrates the consistency and reliability of the intelligent algorithm.
