Human-Object Interaction (HOI) detection is a challenging task of localizing human-object pairs and recognizing their interactions. Recently, DETR-based architectures have shown strong performance for HOI detection, yet they still face two fundamental challenges: suboptimal decoder design and severe long-tailed class distributions. Existing methods typically follow one of two decoder paradigms, either cascaded or layer-wise aligned, both of which are limited in their ability to capture rich instance-level features across decoder layers. Moreover, current models often struggle to recognize rare HOI categories due to extreme class imbalance. To address these issues, we propose DECI-Net, a novel framework that rethinks both decoder structure and class imbalance learning. First, we introduce a Dynamic Layer Fusion mechanism that enables each interaction decoder layer to aggregate instance decoder outputs from multiple layers based on a learnable distance-aware weighting scheme. This design facilitates more flexible and semantically enriched feature interaction. Second, we propose a Class-Specific Adaptive Loss, which dynamically adjusts the contribution of each class during training by assigning class-wise temperatures, effectively alleviating the dominance of head categories. Experiments on the HICO-DET benchmark demonstrate the effectiveness of our method, with particularly notable improvements on tail classes.