Lip-sync deepfakes, which only modify the mouth region to match separate audio tracks, have become increasingly realistic and temporally coherent with recent advances in generative models, making them difficult to perceive and challenging for automated detection. As a result, we propose a generalizable lip-sync video detection framework that jointly analyzes word-level semantic consistency and fine-grained phoneme-level alignment between video and audio. Our approach first compares transcripts produced by a lip-reading model and an automatic speech recognition model to identify overt semantic mismatches. Then, a Temporal Inconsistency-Sensitive Alignment Network (TISAN) is introduced to detect subtle phoneme-level misalignments using cross-modal attention. TISAN leverages frozen pre-trained visual encoders and is trained only on real data with detail-aware temporal augmentations, eliminating the need for synthetic forgeries and enabling robust generalization. Extensive experiments on FakeAVCeleb and DF-40 benchmarks demonstrate that our method achieves state-of-the-art accuracy and maintains high detection performance across different datasets and against unseen, highly realistic lip-sync manipulations. Ablation and alignment studies further verify the effectiveness of each component of our design.