Object detection has achieved remarkable progress, yet its efficacy undergoes substantial deterioration in challenging or adverse environmental conditions. Current domain adaptation object detection (DAOD) methodologies predominantly concentrate on transfer models to acclimate to the target domain, but neglecting the potential erosion in detection performance within the source domains. This narrow focus can undermine the comprehensive robustness and adaptability of the detection systems. We propose a simple but efficient method called Environment-Independent Fusion YOLO (EIF-YOLO) to tackle this issue. Our method focuses on extracting and fusing environment-independent features to enable accurate detection across different domains. We have reused the original feature extractor while preserving all its parameters and optimizing it by mixing data from the source and target domains. To encourage the extraction of object-related features, we introduce multi-layer perceptual regularization to align the features from the original feature extractor. Additionally, we introduce a domain-adaptive fusion that merges features from different domains while minimizing interference with the original data features. Experimental results show that our method surpasses existing foggy and low-light detection approaches while maintaining excellent source domain performance.