With the growing demand for multimedia experiences, accurately assessing audio perceptual quality has become increasingly important. Traditional evaluation methods are often influenced by cognitive bias and fail to fully capture authentic human perception. To address this limitation, this study proposes a brain-inspired multimodal fusion model for perceptual audio quality assessment, aiming to explore how different levels of audio distortion affect human perception. Electroencephalography (EEG) data were collected from subjects exposed to audio stimuli with varying distortion levels, forming a dedicated audio–EEG dataset. Event-Related Potential (ERP) and Mean Opinion Score (MOS) analyses were conducted to validate the relationship between neural responses and perceptual evaluation, confirming the dataset’s reliability. Based on these findings, a multimodal fusion network with a cross-attention mechanism was designed to align audio and EEG features for quality prediction. Experimental results demonstrate that the proposed method outperforms several state-of-the-art approaches and provides a more objective and neurophysiologically grounded framework for audio quality assessment.