Federated Learning (FL) for time-series data has emerged as a promising paradigm to train Machine Learning (ML) models across institutions without sharing data, which is essential in sensitive healthcare settings. Unfortunately, federated models share the black-box characteristics of conventional ML models. While methods such as TimeSHAP provide feature-, event-, and cell-wise explanations for centralized time-series models, their use in FL is limited because they depend on a background dataset that reflects the joint training distribution. As data in FL is typically not identically distributed, clients do not hold such representative datasets, ultimately limiting the quality of explanations they can produce locally. To overcome this limitation, we introduce TimeSHAP–FL, which facilitates a federated, differentially private Generative Adversarial Network (GAN) to synthesize a representative background dataset, which is subsequently incorporated during client-side explanation generation. Our results (1) demonstrate the applicability of TimeSHAP–FL for the task of sepsis onset prediction, (2) indicate improved explanation quality (measured as Spearman Correlation Score) through the combination of local and synthesized background datasets, and (3) account for possible privacy-leakage through the GAN-based data synthesis.