Real-time RGB-D (RGB and depth) semantic segmentation is crucial for dynamic scene understanding. However, processing RGB and depth information requires a substantial number of parameters, resulting in increases the inference time. Although recent efforts have been made to speed up model inference, two critical challenges still hinder effective balancing between inference speed and accuracy: (1) oversimplified network architecture impairing feature extraction capabilities, and (2) inefficient multi-level feature utilization. To overcome those challenges, we introduce a novel real-time RGB-D semantic segmentation model, named Feature Enhancement and Multi-level Interaction Network (FEMINet). The feature enhancement component of FEMINet comprises a Feature Enhanced Module Based on Pooling Attention (PA-FEM) and a Atrous Spatial Pyramid Pooling Based on Residual Connection (RASPP). The PA-FEM can effectively target and strengthen critical feature regions through pooling attention, significantly enhancing feature extraction performance. The RASPP captures multi-scale semantic information through atrous convolutions with residual connections, improving the feature extraction ability for objects of different sizes. Meanwhile, for efficient utilization of multi-layer features, we introduce a Feature Interaction Module Based on Multi-level Semantic Upsampling (MSU-FIM). This module could effectively guide the reconstruction of low-level features through progressive feature interaction. The FEMINet demonstrates competitive results with 52.9% mIoU in NYU Depth v2 and 49.1% mIoU in SUN RGB-D. In particular, FEMINet achieves an inference speed of 51 FPS on RTX3090, demonstrating the balance between segmentation accuracy and inference speed.