In recent years, frequency-domain modeling has emerged as a popular approach for time series forecasting due to its ability to capture recurring patterns. However, classical Fourier-based techniques often face difficulties in representing structural relationships and dynamic variations within non-stationary multivariate sequences, leading to spectral confusion across both temporal and channel dimensions. To address this issue, we introduce SpecMixer, a frequency-aware forecasting framework that jointly alleviates spectral confusion from the perspectives of local temporal dynamics and inter-channel structures. The framework incorporates an adaptive windowed Short-Time Fourier Transform, a Locality-Sensitive Hashing based channel grouping strategy, and a Dual-Bucket Attention mechanism, which together enhance local spectral resolution and strengthen cross-channel dependency modeling. Comprehensive experiments on four benchmark datasets—ETT, Weather, Traffic, and Electricity—demonstrate that SpecMixer consistently surpasses competitive baselines in medium- and long-term forecasting tasks, achieving improved accuracy and robustness. These findings highlight the significance of mitigating spectral confusion across time and channel dimensions and establish a new foundation for frequency-domain modeling of complex multivariate time series.