Indoor scene understanding is critical for enabling effective interactions between robots, humans, and their environments in AI-driven service applications. Real-time semantic segmentation on RGB-D data forms the backbone of this understanding. Although encoder design has been extensively studied, developing compact and robust decoders—equally essential for downstream segmentation tasks—remains underexplored. Conventional decoders often struggle to adapt to the diverse geometries of indoor objects. To overcome this limitation, we introduce the progressive Accumulated Receptive Field Decoder (TARD), a compact yet powerful module that improves the flexibility and resilience of sampling operations during segmentation map reconstruction. TARD is composed of TARD blocks, which employ Bayesian inference and Variational Autoencoders (VAEs) to generate hierarchical, fine-grained offsets based on the expected value and noise-influenced variance of input feature distributions. These offsets enhance geometric adaptation and improve resistance to noise. Furthermore, we present TARDFormer, a novel network that integrates TARD to effectively extract and fuse features from both RGB and depth modalities. By processing multi-level features through TARD, TARDFormer significantly boosts real-time semantic segmentation accuracy. It sets new state-of-the-art performance, achieving 55.9% mIoU on NYUv2 and 50.0% mIoU on SUNRGBD, while running at 80 FPS on an RTX 3090—striking an optimal balance between high accuracy and real-time processing. The code and models are publicly available at https://github.com/TARDFormer/TARD .