<p>Recognizing human emotion from conversations is a key challenge in affective computing, but performance is often hindered by speaker-specific vocal characteristics that obscure underlying emotional content. This limitation is significant because it prevents models from generalizing effectively to new, unseen speakers, which is crucial for real-world applications. This paper tackles this problem by proposing a novel multimodal emotion recognition model that separates speaker identity from emotional features before fusing information from speech and text. Our method first employs a bottlenecked autoencoder to learn a disentangled speech emotion representation, effectively filtering out speaker-specific traits. This purified speech representation is then integrated with a corresponding text representation using a cross-attention mechanism, which learns the complex interplay between the two modalities. We evaluate our model on the IEMOCAP dataset for four-class emotion classification. Experimental results demonstrate the superiority of our approach, as it achieves a new state-of-the-art unweighted accuracy (UA) of 73.83% and weighted accuracy (WA) of 74.65%. These findings underscore that explicitly disentangling speaker-dependent features is a critical and highly effective strategy for building more robust and accurate emotion recognition systems.</p>