Self-supervised pre-training has shown great promise for time series analysis by learning from vast unlabeled data. However, existing methods, particularly those based on masked-reconstruction, often fail to extract comprehensive representations because they model temporal and frequency information in a siloed manner. To address this gap, we introduce DuetFormer, a novel framework for self-supervised feature extraction, built upon a Dual-Channel Time-Frequency Network (DTN) that concurrently processes time and frequency features. At its core is a time-frequency masked-reconstruction strategy, which compels the model to collaboratively learn from both domains by reconstructing jointly masked signals. To ensure representational coherence and synergy, we introduce two key components: a Cross-Domain Adversarial Mechanism (CAM) that aligns feature distributions to learn domain-invariant characteristics, and a Time-Frequency Interactive Comparative Learning (TIL) module that adaptively selects and fuses these representations using an interactive gated unit and a contrastive objective. Extensive experiments on a wide range of benchmarks demonstrate that DuetFormer achieves state-of-the-art performance on both classification and regression tasks, validating the effectiveness of our unified time-frequency approach.