Accurate forecasting of climate and energy trends remains a complex challenge due to the nonlinear and multivariate nature of time series data. In recent years, advances in language models have opened new possibilities for extracting rich contextual information from structured data sets. By converting time series data into natural language-like representations, it becomes possible to leverage the deep semantic understanding of models like BERT to uncover hidden patterns that traditional methods often overlook. In this work, two hybrid are used as forecasting approaches that integrate BERT-based contextual embeddings with LightGBM, a gradient boosting algorithm, to enhance predictive performance. Applied to temperature forecasting using the Jena Climate dataset, this approach achieved a 22–28% reduction in prediction error compared to conventional ARIMA models. Similarly, in forecasting electricity consumption with the LD2011_2014 dataset, the proposed model attains a strong \(R^2\) score of 0.9549. These results suggest that language model embeddings can effectively capture latent temporal semantics, leading to more accurate and interpretable time series forecasts.