In clinical applications, X-Ray technology plays a crucial role in noninvasive examinations like mammography, providing essential anatomical information about patients. However, the inherent radiation risk associated with X-Ray procedures raises significant concerns. X-Ray reconstruction is crucial in medical imaging for creating detailed visual representations of internal structures, and facilitating diagnosis and treatment without invasive procedures. Recent advancements in deep learning (DL) have shown promise in X-Ray reconstruction. Nevertheless, conventional DL methods often necessitate the centralized aggregation of substantial large datasets for training, following specific scanning protocols. This requirement results in notable domain shifts and privacy issues. To address these challenges, we introduce the Hierarchical Framework-based Federated Learning method (HF-Fed) for customized X-Ray Imaging. HF-Fed addresses the challenges in X-Ray imaging optimization by decomposing the problem into two components: local data adaptation and holistic X-Ray Imaging. It employs a hospital-specific hierarchical framework and a shared common imaging network called Network of Networks (NoN) for these tasks. The emphasis of the NoN is on acquiring stable features from a variety of data distributions. A hierarchical hypernetwork extracts domain-specific hyperparameters, conditioning the NoN for customized X-Ray reconstruction. Experimental results demonstrate HF-Fed’s competitive performance, offering a promising solution for enhancing X-Ray imaging without the need for data sharing. This study significantly contributes to the evolving body of literature on the potential advantages of federated learning in the healthcare sector. It offers valuable insights for policymakers and healthcare providers holistically. The source code and pre-trained HF-Fed model is available at this link .