This study introduces and assesses a cancer-immune system model using the \(\psi \) -Caputo fractional derivative, a sophisticated operator that adds an auxiliary function \(\psi (t)\) to the traditional Caputo formulation. This generalization provides a more accurate biological complexity characterization by capturing heredity and memory effects in tumor-immune dynamics with a broader and more flexible mathematical framework. The three-dimensional nonlinear fractional model of tumor, healthy tissue, and effector immune cell populations shows chaotic oscillations and other dynamics. Stability analysis of the system’s equilibrium points finds many biologically tolerable steady states for tumor-free, tumor-only, and tumor-immune cohabitation. An effective and convergent iterative procedure using successive substitutions approximates the \(\psi \) -Caputo fractional initial-value problem (IVP) for precise numerical analysis of this generalized model. Numerical tests indicate that the proposed algorithm reproduces known chaotic regimes at specified parameter values and accurately captures the chaotic cancer-immune system’s complicated transient and asymptotic dynamics. The paper also uses the Pontryagin’s Maximum Principle (PMP) to solve a fractional optimal control problem (OCP) in \(\psi \) -Caputo systems, reducing tumor cell concentration and suppressing chaotic oscillations. The coupled state-adjoint system is numerically solved using an extended forward–backward scheme and the proposed iterative approach. As a result, the optimal control reduces tumor cell populations and improves immunological balance by suppressing chaos and stabilizing system dynamics, according to simulations. Consequently, this research introduces a new framework for modeling, simulating, and manipulating complex chaotic cancer-immune interactions using \(\psi \) -Caputo fractional calculus. Fractional dynamics captures long-memory effects, the numerical method is reliable, and the fractional optimal control scheme can reduce biological system chaos effectively.