<p>The aim of this research is to explore the adaptation of fuzzy time series (FTS) modeling and forecasting for dynamically evolving non-stationary data. It is proposed that fuzzy time series modeling, with fuzzy logical relationships (FLRs) predicted by deep learning, and hyperparameters (fuzzy order and length of intervals) defined by swarm intelligence, can effectively forecast non-stationary time series data. The global outbreak of the COVID-19 pandemic highlights the need for accurate time series forecasting models that can adapt to multiple successive waves of the pandemic caused by dynamically evolving variants of the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) having distinctive spread patterns. Therefore, in this study, a hybrid time series forecasting model integrating high-order FTS, context-augmented variants of the long short-term memory network (LSTM), and particle swarm optimization (PSO), is proposed for accurate forecasting of COVID-19 cases associated with multiple COVID-19 waves. Attention and convolution mechanisms are explored for context augmentation in LSTM. The proposed hybrid model is evaluated on five different datasets of COVID-19 confirmed cases in USA, UK, India, Russia and Italy, in the duration of June 1, 2020 to April 15, 2022, encompassing multiple COVID-19 waves. The model forecasts are compared with five state-of-the-art time series forecasting models using five different performance metrics. Experimental results prove that the hybrid of FTS, attention-bidirectional-LSTM, and PSO (FTS+PSO+Attention-Bi-LSTM) performs consistently best for all countries. Nemenyi statistical significance test verifies that FTS+PSO+Attention-Bi-LSTM is the leading model for time series forecasting of the COVID-19 pandemic with 95% confidence.</p>