Temporal Action Detection (TAD) aims to localize and classify actions in untrimmed videos by modeling long-range dependencies and filtering background noise. While recent video Transformers (ViTs) have advanced the field, snippet-based processing often disrupts temporal continuity and leads to over-smoothed features. Moreover, transient but critical temporal cues—especially peak activation patterns around action boundaries—are frequently underexplored, limiting precise localization. To tackle these challenges, we propose PACTFormer, a unified framework with two key components: (1) the Peak-Aware Cross-Temporal (PACT) module, which explicitly models salient motion across snippets by aggregating peak activations as global cues, enabling long-range reasoning; and (2) the Scalable-Granularity Local-Global (SGLG) module, a lightweight convolutional neck that captures fine-to-coarse temporal context to improve feature diversity and mitigate rank degradation. Together, these modules enhance both temporal coherence and feature discriminability. PACTFormer achieves state-of-the-art mAP scores of 71.9% on THUMOS14 and 39.02% on ActivityNet-1.3, demonstrating strong performance on long-range temporal modeling and action localization.