Advances in deep generative models have made realistic audio-visual deepfakes increasingly common, leading to growing concerns about media authenticity. Although recent methods utilize the synchrony between speech and facial motion for detection, they often fail in real-world conditions where visual quality is degraded or forgeries display highly accurate audio-visual alignment. In this paper, we propose QM-AVAlign, a novel quality-aware and multi-scale audio-visual alignment framework for robust video deepfake detection. QM-AVAlign employs a visual quality assessment module to dynamically generate spatial reliability masks for each frame, emphasizing high-confidence visual cues while suppressing unreliable regions. Leveraging these quality-weighted visual features, we introduce a multi-scale cross-modal fusion strategy to simultaneously model global synchrony and fine-grained, physiologically grounded relationships between speech and facial expressions. Additionally, an uncertainty calibration loss is introduced, guiding the model to produce higher uncertainty scores when faced with ambiguous or low-quality samples. Extensive experiments on multiple public benchmarks demonstrate that QM-AVAlign achieves superior robustness and generalization under challenging conditions with visual impairments and advanced forgery techniques.