This study investigates the impact of incorporating multimodal affective signals into foundation models, focusing on whether such integration enhances their ability to represent emotional content. We compare task-agnostic foundation models with those pretrained specifically on affective data to assess the benefits of affective pretraining. Additionally, we examine how multimodal pretraining influences the robustness and generalizability of affective representations, emphasizing the role of cross-modal learning. In particular, we explore how pretraining on different combinations of modalities—audio-text, audio-video, video-text, and all three—affects the quality of representations learned for each modality, especially when certain modalities are absent during downstream emotion recognition tasks. Our experiments across multiple benchmarks demonstrate that incorporating additional modalities during pretraining consistently improves performance, even when those modalities are unavailable at inference time. Notably, models pretrained on all three modalities achieve the highest generalizability, showing significant gains in both low-resource settings and related tasks such as humor detection. These findings highlight the value of cross-modal knowledge transfer and underscore the importance of modality-rich pretraining for building robust, adaptable emotion-aware systems.