<p>The generalized autoregressive conditional heteroscedasticity, GARCH, models are the most popular models for capturing time-varying symmetric volatility in financial and economic time series data. Most of the theoretical results on GARCH models assume a constant or linear conditional mean. The smooth transition autoregressive, STAR, models allow for nonlinear behavior in the time series. The STAR-GARCH model combines regime switching and volatility models, with the regime switching in the mean equation and volatility in the variance equation. The estimation of smooth transition autoregressive models with GARCH errors, STAR-GARCH, are not entirely straightforward. Therefore, likelihood functions are appratimed using numerical methods. It is worth mentioning that the convergence of the maximum likelihood estimator for STAR-GARCH models is sensitive to initial values. This paper investigates modified maximum likelihood estimators of the parameters of smooth transition autoregressive models with GARCH errors and asymptotic distribution of modified maximum likelihood estimators. As a result, we can select optimal models based on Vuong’s test. A simulation study leads strong support to the results presented in the paper. We studied Wilshire 5000 total market full cap index dataset and selected an optimal model for this data based on the theoretical results.</p>