In recent years, significant advancements have been made in speech recognition technology. The conformer encoder and attention-rescoring decoding method within the portable Wenet toolkit have garnered considerable attention. However, the conformer encoder in Wenet has limitations, such as lacking inter-layer skip connections. Additionally, while the attention-rescoring decoding method improves recognition accuracy, errors from CTC beam search decoding may accumulate during subsequent attention decoding, affecting the final recognition results. We propose an Enhancing Conformer Models with Innovative Scoring Matrices (ECMISM) model to address these issues. We optimized the attention-rescoring decoding method by introducing a novel Relationship Calculation Module (RCM). This module aims to mitigate error accumulation in attention-rescoring decoding methods, thereby reducing the impact of CTC decoding errors on subsequent attention recovery. Additionally, we introduce a Skip Fusion Module (SFM) to integrate shallow and deep features. This addresses the limitation of the Conformer encoder’s lack of inter-layer skip connections, enhancing the model’s capability to capture and utilize contextual information effectively. The experimental results demonstrate that our approach has achieved outstanding performance, particularly on the relatively small Uyghur dataset. Compared to the baseline models, our method reduces character error rates by 0.03%, 0.35%, and 0.44% on the Aishell1, Primewords, and ST datasets, respectively. On the General Speech 16.1 Uyghur dataset, our approach achieves a word error rate of 4.19%, which is 1.56% lower than the baseline model.