Cross-cultural humor detection presents a complex challenge in multimodal sentiment analysis due to its nuanced nature and cultural variability. This paper introduces Residual Attention Multimodal Model (RAMM), a novel approach for multimodal humor detection that effectively fuses audio, visual, and textual features. RAMM is evaluated on the Passau Spontaneous Football Coach Humor (Passau-SFCH) dataset, which is part of the 2024 MuSe-Humor sub-challenge. The proposed model employs residual connections and attention mechanisms to capture intricate cross-modal interactions. Our experimental results show that RAMM outperforms the unimodal approaches, with the best configuration achieving an AUC of 0.9298 on the development dataset and 0.8631 on the test dataset. These results highlight the effectiveness of our multimodal fusion approach in capturing the complexities of spontaneous humor in different cultural contexts.