Integrating audio encoders with LLMs has enabled models to process audio, enhancing speech-to-text tasks including automatic speech recognition (ASR) and automatic speech translation (AST). However, these methods often overlook language adaptation in multilingual settings, relying on multilingual data without adequately addressing language differences. To address this gap, we propose the Ideal-LLM model, which employs dual multilingual encoders to enrich language features and uses a language-adapted connector to target each language. By leveraging the complementary strengths of Whisper and MMS encoders, our approach ensures richer multilingual representations. Additionally, the connector enhances modal transformation via a weight selector tailored for each language. Experimental results demonstrate that Ideal-LLM improves ASR performance, achieving a 32.6% relative reduction in word error rates compared to the standard speech encoder integrated with LLMs and yields an average BLEU score of 36.78 for AST.