Predicting drug-target affinity (DTA) is key to modern drug discovery and a benchmark task for graph algorithms, demanding heterogeneous signal integration. However, existing models use single-modality inputs and simplistic fusion, leading to fragmented and misaligned representations. To address this, we propose MMCA-DTA, a Multi-Modal Contrastive Alignment framework for graph representation learning. MMCA-DTA constructs two complementary graph-based views: an Intrinsic Structural View capturing molecular topology via a Graph Convolutional Network (GCN), and an Extrinsic Contextual View. To generate this contextual view, we construct a heterogeneous drug-target interaction (DTI) network, where initial node features are pre-computed Morgan fingerprints for drugs and pre-trained ESM-2 embeddings for targets. A second GCN then learns the contextual representations by propagating information across this DTI graph. The framework is optimized through a multi-task objective combining affinity regression and a multi-positive contrastive loss that enforces consistency between the two views. Experiments on our curated large-scale BindingDB dataset, as well as the Davis and KIBA benchmarks, confirm that MMCA-DTA achieves state-of-the-art results across multiple regression metrics, demonstrating both strong scalability and generalization. MMCA-DTA provides a generalizable paradigm for multi-modal graph alignment, harmonizing structural and contextual representations in complex networks. Code and data are available at https://github.com/imustu/MMCA-DTA .