This work introduces a physics-informed Transformer framework for unsupervised anomaly detection in gamma-ray spectra, a key task in nuclear security. The proposed approach, that is benchmarked against a classical Kullback–Leibler divergence-based Nonnegative Matrix Factorization (KL-NMF) model, implements a novel Transformer-based NMF decoder that integrates self-attention mechanisms with physical constraints derived from Poisson photon statistics and detector response behavior. Using a dataset comprising 18,000 background spectra augmented with 2,000 synthetically injected anomalies, both models are trained and evaluated under identical reconstruction-based scoring criteria. The Transformer model demonstrates markedly superior detection performance (ROC-AUC = 0.6243 vs. 0.5685; PR-AUC = 0.7367 vs. 0.6706) and achieves 41.6% higher true-positive rates at low false-alarm thresholds ( \(10^{-3}\) ). By embedding domain physics within an attention-driven spectral representation, the proposed architecture captures nonlinear dependencies, enhances noise robustness, and maintains interpretability through a positive latent basis. These results position physics-informed Transformer architectures as a promising foundation for interpretable, high-fidelity radiation anomaly detection in next-generation monitoring systems.