Image Quality Assessment (IQA), especially No-Reference Image Quality Assessment (NR-IQA), is a crucial yet challenging task in image processing. Although deep learning-based methods have achieved remarkable progress, they often struggle with the domain gap between synthetic and authentic distortions, which limits their generalization capability. Moreover, effectively capturing multi-scale distortion features, particularly high-frequency details and global structural information, remains a persistent challenge. To tackle these issues, we propose CLEST-IQA, a novel NR-IQA model based on contrastive learning and the Swin Transformer. Our approach employs a cross-domain contrastive learning framework, where synthetic data is used in an auxiliary task to learn distortion-specific features, while authentic data enhances generalization ability in real-world distorted scenes through an instance discrimination task. Additionally, we utilize the hierarchical structure of the Swin Transformer to extract multi-scale features, which are dynamically fused with contrastive learning representations via a hierarchical attention fusion module. This module adaptively balances global structural information and local details, ensuring robust performance across diverse distortion types. Extensive experiments on seven widely-used datasets have demonstrated that CLEST-IQA surpasses state-of-the-art methods, particularly in cross-domain scenarios.