Multimodal image segmentation utilizes a variety of modality images with RGB, infrared, polarization, etc. Unfortunately, the mainstream focus on digital modality fusion leads to the cost of computing abundant information and increased model size. To selectively capture meaningful information and compress model size, we introduce a novel framework for hardware level modality fusion (Our code is available at https://github.com/eki-1996/Deep_Hardware_Modality_Fusion . Our hardware modality fusion module simulates a captured image with fused modality on hardware in two strategies: frame coding and pixel coding. It also enables a single encoder model structure and significantly reduces model size. This modality fused image makes the encoder eliminate redundant information to extract the necessary information for downstream tasks. We evaluated our methods on two multimodal segmentation datasets that contain different modalities and got comparable performance on both datasets compared to baseline methods, which explore different digital modality fusion strategies. Our proposed method outperforms baseline methods on evaluated datasets and tasks with compressed information passed to the segmentation model.