Recognizing emotion from voice is essential for adaptive social behavior, voice-based emotion recognition methods often struggle with accuracy due to the semantic similarity between words with different emotional meanings. This study aims to enhance emotion recognition accuracy by integrating both pattern and semantic features. For pattern features, we employ Convolutional Neural Networks (CNNs) to detect patterns in voice data and Long Short-Term Memory (LSTM) networks to model the relationships between these patterns and emotions. To capture semantic features, we utilize pre-trained BERT to provide context-based semantic representations of text converted from voice. These pattern and semantic features are fused using an attention mechanism through a specially designed attention fusion module to assess their correlation. We evaluate this combined method using the MELD and SLUE datasets, achieving a significant accuracy improvement to 71%, surpassing that of non-semantic feature-based approaches.