Multi-task image restoration typically requires separate models for different degradation types, limiting practical deployment in real-world scenarios where multiple degradations coexist. This paper proposes MFIARNet (Multi-Factor Image Adaptive Restoration Network), a unified framework that simultaneously handles denoising, deraining, dehazing, and deblurring without explicit degradation type identification. The architecture incorporates two key innovations: the Blur-Aware Contrastive Encoding Module (BACEM) for degradation pattern recognition through contrastive learning, and the Adaptive Multi-Task Restoration Module (AMTRM) for dynamic feature fusion and task-specific adaptation. Extensive experiments demonstrate superior performance across benchmark datasets, with MFIARNet achieving 34.65 dB PSNR on Urban100 (denoising σ = 15), 36.92 dB PSNR on Rain100L (deraining), 28.08 dB PSNR on SOTS (dehazing), and 29.28 dB PSNR on GoPro (deblurring), yielding an overall average of 31.63 dB PSNR. This substantially outperforms existing methods including LPNet (28.15 dB) and FDGAN (29.47 dB), while achieving competitive results with Restormer (31.89 dB). The unified architecture eliminates the need for degradation classification preprocessing, significantly reducing system complexity while maintaining superior restoration quality.