Accurate assessment of the flood damage and its severity estimation is essential for effective disaster management and related reconstruction works. In our study, we propose a novel approach for estimating the severity of flood damage from aerial scene images. We introduce a fusion network architecture that leverages both RGB and generated pseudo thermal image modalities. Our approach is based on training a CNN head as U-Net model to perform semantic segmentation of images from flood scenes. We show that the feature maps extracted from multimodal data, helps to improve accuracy even though one of the modalities is pseudo generated via CycleGAN. These feature maps are fed into a custom fully connected network for regression, predicting the severity level of flood damage. Our use of CycleGAN to generate thermal images from RGB images, providing additional input modalities for our network. Our approach significantly outperforms baseline methods, showcasing the effectiveness of leveraging multiple modalities for flood damage severity estimation. The results from our regression network show that our fused network outperforms conventional approaches. Our method achieves 0.053 MAE and 0.008 MSE, indicating a substantial enhancement of performance compared to baseline methods. These results show the importance of multimodal fusion via pseudo modality generation, which also offers valuable insights in flood damage assessment.