Recent research has shown that large language model (LLM) can be effectively used for real-world time series forecasting due to their strong natural language understanding capabilities. However, these approaches face a fundamental challenge that LLM operate on discrete tokens, while time series data is continuous. Meanwhile, recent works leverage pre-trained visual masked autoencoders (visual MAE) to construct time series forecasting foundation models. Nevertheless, converting time series into images disrupts the critical temporal dependencies of time series. Additionally, it is intuitive that different sampling scale time series exhibit distinct patterns, where the microscopic information is reflected in the fine scale, while the macroscopic information is reflected in the coarse scale. Based on these observations, we first generate multi-scale time series through downsampling to capture diverse temporal patterns, then we design a novel dual-modality encoding framework for long-term time series forecasting, consisting of an LLM encoding branch for discrete semantic reasoning and a visual MAE encoding branch for continuous representation learning. To effectively leverage the complementary strengths of both LLM encoding branch and visual MAE encoding branch, we propose a mixture of multi-scale multi-modal experts (M3E) to fuse features from the LLM with features from the visual MAE. Furthermore, M3E adaptively selects key-scale features from multi-scale features from LLM and visual MAE respectively to reduce computational costs, and facilitates multi-scale features interaction, which enables the capture of both short-term details and long-term patterns. Extensive experiments on six real-world datasets demonstrate M3E is a powerful time series model, outperforming state-of-the-art methods.