Transmission line defect detection plays a critical role in ensuring the safe and stable operation of power systems. However, this task faces significant challenges due to the randomness of aerial image capture angles, large variations in object scales, complex backgrounds, and weak features of small objects. To address these issues, we propose FFTA-Net, a novel defect detection network based on Frequency-domain Feature Fusion and Temporal-domain Feature Alignment. In the frequency domain, we perform multi-scale fusion on low-level features from the FPN outputs to enhance the representation of small object features. Meanwhile, a specially designed align-loss function is introduced to align multi-level feature representations in the temporal domain, improving multi-scale consistency and detection accuracy. Additionally, a foreground-aware mechanism is integrated to strengthen foreground features while suppressing background interference, enhancing robustness under complex environments. Extensive experiments on a custom transmission line defect dataset demonstrate that FFTA-Net significantly improves detection accuracy and generalizes well across different object detection models.