Accurate synthesis of patient-specific chest X-ray (CXR) images that reflect temporal disease progression remains challenging due to the complex interplay of multimodal longitudinal data. We introduce PAX-Diff, a multimodal progression-aware chest X-ray generation framework built on a diffusion model, which synthesizes future medical images by tracking temporal changes within patients’ historical sequences of X-rays and radiology reports. The key is the proposed cross-modal progression-aware conditioning net, consisting of two core components: an intra-visit multimodal learner to align image-texts, and a cross-visit causal attention module to connect the underlying info globally across all image-text pairs from historical visits. Benefiting from this structure, it provides a conditioning signal that effectively integrates historical information, enabling the controlled generation of future images. By extending next-token prediction to cross-visit feature blocks, PAX-Diff explicitly models temporal dependencies across clinical visits, thus provides a predicted multimodal representation of the next visit with causality. Additionally, we propose a hierarchical condition alignment through global cosine similarity and local-level perceptual alignment to refine the model’s training process, enhancing its ability to produce accurate images with clinical semantic consistency.