In smoky environments, the fusion of infrared and visible images leverages the smoke-penetrating property of infrared imaging to remove smoke from visible images. However, due to the lower exposure of infrared images, the suppression of visible images in the smoke region of the fused image can result in unnatural transitions between smoke and non-smoke regions, leading to pronounced boundary artifacts. To address this issue, we propose a Multi-slice Poisson Edge Softening Algorithm (MPEA) that uses edge knowledge to ensure a smooth and natural transition between smoke and non-smoke regions in the fused image. Furthermore, we introduce the quadrangle attention transformer module (QATransformer) as a decoder to preserve the detailed textures in non-smoke regions during feature reconstruction. Finally, we integrate these components into the proposed smoke removal fusion network, enabling edge-knowledge-driven adaptive fusion of infrared and visible information. Comparative and ablation experiments demonstrate the effectiveness of the proposed method.