In this article, we analyze a nonlinear time-fractional-order epidemic model of smoking via Caputo-type nonlocal fractional differential operator due to its mathematical attribute of memory effects. This study considers the interaction of various demographic classes such as susceptible smokers X(t), snuffing category \({{H}^{1}(t)}\) , irregular smokers \({{H}^{2}(t)}\) , regular smokers Y(t) and quit smokers Z(t). A significant aspect of the model is the incorporation of relapse from the regular smoker class back into the snuffing category, as well as a feedback loop between regular smokers and tobacco users, which illustrates the repetitive and compulsive nature of tobacco consumption. The existence and uniqueness of the fractional model’s solution are demonstrated using fixed-point theory. Basic reproduction number \({R}_0\) and equilibrium points are computed. The local and global stability of the smoking-free and smoking-present equilibrium is examined. A sensitivity analysis of the basic reproduction number \((R_0)\) is conducted to determine the key parameters that influence smoking prevalence the most. The findings reveal that the transition rate into smoking and the recruitment rate exert the greatest effect on \(R_0\) , highlighting the necessity for focused prevention strategies. Moreover, we obtain a solution of the model via the Fractional Variational Iteration Method (FVIM) and analyze the convergence of its solution. Numerical simulation for the considered nonlinear smoking model is discussed for different fractional order \(\alpha\) at initial approximations. The outcomes are illustrated via figures which demonstrate population variation and asymptotic behavior. The study shows that the model with fractional order \(\alpha = 0.8\) enhances the model’s alignment with empirical data, highlighting the importance of fractional calculus in accurately representing the intricate dynamics of complex systems. We observe that a change in the fractional order directly affects the dynamics of the nonlinear smoking model.